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Abstract 

Throughout  Operations  Iraqi  Freedom  and  Enduring  Freedom,  the  Department  of 
Defense  (DoD)  faced  challenges  not  experienced  in  our  previous  military  operations. 

The  enemy’s  unwavering  dedication  to  the  use  of  improvised  explosive  devices  (IEDs) 
against  the  coalition  forces  continues  to  challenge  the  day-to-day  operations  of  the 
current  war.  The  Joint  Improvised  Explosive  Device  Defeat  Organization ’s  (JIEDDO) 
proposal  solicitation  process  enables  military  and  non-military  organizations  to  request 
funding  for  the  development  of  Counter-Improvised  Explosive  Device  (C-IED)  projects. 

Decision  Analysis  (DA)  methodology  serves  as  a  tool  to  assist  the  decision  maker 
(DM)  in  making  an  informed  decision.  This  research  applies  Value  Focused  Thinking 
(VFT),  a  specific  methodology  within  DA,  to  the  JIEDDO  proposal  selection  process  in 
order  to  assist  DMs  in  filtering  out  proposals  that  do  not  meet  desired  C-IED  objectives. 

This  research  evaluated  the  validity  of  the  previously  developed  JIEDDO 
Proposal  Value  model  to  address  the  following  questions:  Does  the  value  model 
adequately  reflect  JIEDDO ’s  decision  process;  and  secondly,  given  the  dynamic 
environment  of  the  current  war,  how  confident  can  we  be  in  the  model ’s  ability  to 
continually  and  effectively  screen  proposals  based  JIEDDO ’s  current  values?  The 
author  utilizes  multivariate  techniques  to  investigate  JIEDDO ’s  ability  to  make  consistent 
decisions  within  their  proposal  evaluation  process.  Once  it  has  been  determined  that  the 
model  effectively  screens  proposals,  it  is  possible  to  proceed  with  the  second  question. 

By  consolidating  and  applying  n-way  sensitivity  analysis  techniques  the  author  proposes 
a  consistent  sensitivity  analysis  image  profiling  technique. 
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ROBUST  SENSITIVITY  ANALYSIS  FOR  THE  JOINT  IMPROVISED  EXPLOSIVE 
DEVICE  DEFEAT  ORGANIZATION  (JIEDDO)  PROPOSAL  SELECTION  MODEL 


I.  Introduction 

Commanders  make  decisions  that  require  conscientious  examination  of  the 
options  under  their  control.  They  rely  on  decision  analysts  to  utilize  the  tools  and 
techniques  thereof  to  aid  their  decision  making  process.  The  recently  proposed  Joint 
Improvised  Explosive  Device  Defeat  Organization  (JIEDDO)  value  model  is  capable  of 
serving  as  a  filtering  tool  for  evaluating  Counter-Improvised  Explosive  Device  (C-IED) 
proposals  to  become  fully  funded  initiatives.  Although  JIEDDO  recognizes  that  using  a 
tool  to  track  a  proposal’s  positive  and  negative  characteristics  and  provide  justifiable 
feedback  to  the  applicant  is  useful,  they  have  expressed  a  desire  for  more  confidence  in 
the  model’s  ability  to  reflect  adequately  the  decision  at  hand.  Much  of  their  willingness 
to  embrace  the  model  lies  in  the  analyst’s  ability  to  demonstrate  that  it  is  a  valid 
representation  of  their  decision  process.  In  order  to  meet  their  request,  it  is  necessary  to 
show  that  the  model  does  in  fact  reflect  the  proposal  evaluation  process  and  that  it  will 
continue  to  do  so  as  the  organization  evaluates  future  proposals.  A  great  deal  of  the 
evaluation  relies  on  the  robustness  of  the  weighted  values  that  are  the  basis  for  evaluating 
a  given  proposal. 

The  architects  of  the  original  JIEDDO  value  model  recognized  the  importance  of 
conducting  sensitivity  analysis  on  the  weights  and  provided  substantial  one-way  and  two- 
way  sensitivity  analysis  to  the  decision  problem.  However,  their  post  model  analysis  did 
not  include  model  weight  adjustments  for  three  or  more  values  simultaneously.  This 
thesis  applies  multivariate  techniques  to  conduct  n-way  sensitivity  analysis  in  order  to 
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aide  decision  makers  from  JIEDDO  and  other  organizations  to  assert  confidently  their 
assessment  of  alternatives.  With  this  knowledge,  they  will  thereby  make  a  decision  fully 
grounded  in  the  values  that  encapsulate  the  issues  under  consideration. 

I.A.  Background 

JIEDDO 

From  October  2003  to  August  2008  the  total  number  of  reported  U.S.  fatalities  in 
Iraq  attributed  to  detonated  IEDs  summed  to  1717  people  (Iraq  Collation  Causualty 
Count).  According  to  the  Congressional  Research  Service,  IEDs  account  for  60  percent 
of  all  U.S.  “combat  casualties  both  killed  and  wounded”  in  Iraq  and  50  percent  in 
Afghanistan  (United  States  Government  Accountability  Office  ,  2008).  In  the  first  three 
years  of  the  war,  the  Secretary  of  Defense  recognized  a  need  for  a  well-established  and 
organized  C-IED  organization. 

In  February  2006,  the  Department  of  Defense  (DoD)  organized  the  joint  C-IED 
efforts  of  the  Joint  Improvised  Explosive  Devise  Defeat  Task  Force  by  establishing  the 
Joint  Improvised  Explosive  Device  Defeat  Organization  (JIEDDO:  About  Us). 

JIEDDO ’s  current  mission  is  “to  lead,  advocate  and  coordinate  all  DoD  actions  in 
support  of  Combatant  Commanders’  and  their  respective  Joint  Task  Forces’  efforts  to 
defeat  improvised  explosive  devices  (IEDs)  as  weapons  of  strategic  influence”  (JIEDDO: 
About  Us). 

In  order  to  remedy  the  current  IED  problem,  JIEDDO  solicits  support  from 
military  and  civilian  communities.  JIEDDO  identifies  C-IED  proposals  that  meet  the 
needs  of  our  current  warfighter,  ensures  the  contribution  of  appropriate  funding  to  these 
initiatives,  and  more  importantly  delivers  an  effective  product  to  implement  in  the  field. 
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JIEDDO’s  current  proposal  solicitation  policy  requires  interested  parties  to  submit  their 
C-IED  proposals  to  the  Broad  Area  Announcement  Infonnation  Delivery  System  (BIDS) 
(JIEDDO:  About  Us).  BIDS  communicates  to  the  public  JIEDDO’s  current  interests  in 
the  way  of  C-IED  capabilities  and  receives  proposals  for  potential  future  C-IED 
initiatives.  Traditionally,  when  a  proposal  is  submitted  to  JIEDDO  for  consideration,  it 
undergoes  an  extensive  evaluation  process  that  involves  a  panel  of  14-16  subject  matter 
experts  (SMEs)  who  assess  a  group  of  proposals  over  a  three-day  time  period  (Mauldin, 
2008).  The  team  of  evaluators  is  unavailable  to  conduct  the  duties  of  their  day-to-day  job 
during  this  time  period.  The  panel  reviews  proposals  based  on  the  characteristics  (or 
variables)  that  each  proposal  claims  it  will  achieve.  Upon  completion,  the  panel 
recommends  to  the  JIEDDO  Director  the  rejection  or  acceptance  of  the  proposal 
(Mauldin,  2008).  Upon  hearing  these  recommendations,  the  Director  reviews 
programmatic  and  resource  implications  for  selecting  the  proposed  C-IED  initiatives  and 
provides  the  stamp  of  approval  for  the  proposal  to  enter  the  first  stage  of  the  acquisition 
process  (Mauldin,  2008). 

House  Armed  Service  Committee  Oversight  &  Investigation  of  JIEDDO 

JIEDDO’s  ability  to  carry  out  its  C-IED  mission  heavily  relies  on  having  the 
fiscal  funds  to  do  so.  Congress  has  recognized  this  need  and  has  been  monetarily 
accommodating.  In  fiscal  year  2007  alone,  Congress  provided  approximately  $4.35 
billion  to  JIEDDO  (United  States  Government  Accountability  Office  ,  2008).  The  influx 
of  monetary  resources  is  expected  to  continue  pending  JIEDDO’s  ability  to  demonstrate 
its  productivity  and  efficiency  within  the  DoD. 
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JIEDDO’s  large  congressionally  appropriated  budget  and  delicate  mission, 
however,  have  made  it  the  Government  Accountability  Office’s  (GAO)  target  for 
investigation.  Over  the  past  two  years,  the  GAO  conducted  audits  to  evaluate  JIEDDO’s 
ability  to  effectively  and  appropriately  carry  out  the  C-IED  mission.  In  their  initial  report 
in  March  2007,  the  GAO  investigated  JIEDDO’s  “management  and  operations”  ability 
(United  States  Government  Accountability  Office  ,  2008).  JIEDDO  was  criticized  for  an 
apparent  “lack  of  a  strategic  plan  and  the  resulting  effects  on  the  development  of  its 
financial  and  human  capital  management  programs”  (United  States  Government 
Accountability  Office  ,  2008).  Approximately  one  year  after  an  initial  review,  in  2008, 
the  GAO  re-attacked  with  focused  efforts  on  JIEDDO’s  financial  management  process 
and  the  organization’s  ability  to  “identify,  record,  track,  and  report”  on  all  employees  to 
include  contractors  (United  States  Government  Accountability  Office  ,  2008). 

In  spite  of  all  the  scrutiny,  JIEDDO  remains  confident  that  they  are  winning  the 
C-IED  fight.  Providing  detailed  proof  of  this  statement,  however,  poses  great  challenges. 
In  addition  to  the  GAO,  the  public  at  large  seeks  substantial  evidence  that  JIEDDO  is 
making  a  substantial  positive  difference  in  the  current  war. 

Decision  Analysis 

Decision  Analysis  (DA)  is  a  field  within  Operations  Research  that  helps  the 
community  at  large  by  aiding  decision  makers  to  make  appropriate  and  informed 
decisions.  More  specifically,  DA  is  used  as  a  means  to  aid  the  decision  maker  (DM)  in 
selecting  the  “best”  alternative  for  a  given  decision  problem.  For  JIEDDO,  the 
conscientious  evaluation  of  C-IED  proposals,  though  important,  requires  careful 
examination  by  numerous  involved  parties  to  ensure  the  selected  proposals  are  qualified. 
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A  decision  analysis  model  provides  a  consistent  systematic  framework  for  proposal 
evaluation  and  decision  justification  (Dawley,  Marentette,  &  Long,  2008). 

To  construct  a  model  that  adequately  encapsulates  the  decision  problem,  it  is 
crucial  that  the  analyst  works  with  the  appropriate  DM  to  identify  all  of  the  values  for  the 
decision  at  hand.  After  identifying  the  values,  it  is  possible  to  detennine  the  measures  by 
which  the  values  will  be  evaluated.  The  analyst  works  with  the  DM  to  detennine  the 
weights  for  each  of  the  identified  values.  The  weights  reflect  the  DM’s  preferences 
within  the  decision  problem. 

For  the  JIEDDO  proposal  evaluation  process,  utilizing  a  decision  model  that 
encapsulates  the  appropriate  DM’s  values  as  a  decision  making  tool  serves  a  dual 
purpose.  First,  it  serves  as  a  filtering  tool  that  allows  senior  leaders,  such  as  those  serving 
on  the  proposal  evaluation  panel,  to  concentrate  their  efforts  on  examining  proposals  that 
have  the  greatest  potential  for  meeting  the  C-IED  mission  requirement  and  implementing 
them  appropriately.  Second,  it  exploits  DA  techniques  to  provide  a  “defensible  and 
repeatable  framework”  to  aid  the  proposal  screening  process  (Dawley,  Marentette,  & 
Long,  2008). 

Sensitivity  Analysis 

A  value  model  produces  a  score  for  each  alternative  using  an  additive  value 
function  calculation  that  is  the  sum  of  the  weighted  values  themselves.  From  these 
scores,  the  DM  identifies  the  best  alternative,  that  which  possesses  characteristics  that 
they  value  most.  Prior  to  taking  action,  however,  it  is  necessary  to  evaluate  the 
sensitivity  of  the  alternatives  to  weight  change  variations. 
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The  weights  reflect  the  DM’s  preferences.  Consequently,  they  are  subjective.  It 
is  important  to  investigate  how  scores  produced  using  the  additive  value  function  are 
affected  by  weight  changes  if  the  DM’s  preferences  are  different  than  those  originally 
solicited.  Thus,  if  the  DM  is  slightly  off  in  their  assessment  of  the  weights,  how 
confident  can  we  be  in  selecting  a  particular  alternative?  Sensitivity  analysis  allows  the 
decision  maker  to  view  how  perturbing  the  weight  for  a  particular  value  affects  the 
decision  outcome. 

Traditionally,  analysts  conduct  one-way  sensitivity  analysis  to  identify  single- 
handedly  which  value  affects  the  decision  problem  most  (Skinner,  1999).  Similarly,  two- 
way  sensitivity  analysis  allows  for  the  alteration  of  two  different  weights  simultaneously 
and  observing  changes  in  the  decision  outcome.  In  most  cases,  one-way  and  two-way 
sensitivity  analysis  provide  a  “screening”  process  for  identifying  which  weights  have  the 
greatest  potential  to  affect  the  decision  problem  (Bauer,  Parnell,  &  Meyers,  1999). 
Although  the  effects  of  altering  one  or  two  weights  are  convenient  analysis  endeavors, 
the  majority  of  real  world  situations  have  uncertainty  in  more  than  two  weights.  From 
this,  we  are  faced  with  the  following  questions:  What  are  the  consequences  of  altering 
three,  four,  five,  or  n  weights  simultaneously?  Will  altering  each  of  the  weights  by  even 
a  small  amount  relative  to  its  original  weight  completely  restructure  the  ranking  of 
alternatives  themselves  (Bauer,  Parnell,  &  Meyers,  1999)?  Addressing  such  questions 
will  allow  analysts  to  gain  insight  into  the  validity  and  robustness  of  value  models  like 
the  proposed  JIEDDO  Proposal  Value  Model.  Once  we  have  adequately  addressed  these 
issues,  it  is  possible  to  determine  whether  further  utilization  of  such  a  value  model  is 
appropriate. 
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I.B.  Problem  Statement 


For  JIEDDO  the  consequences  of  suboptimally  allocating  funds  as  a  result  of 
inconsistent  decisions  has  the  potential  of  leaving  the  warfighter  ill  equipped  to  meet 
mission  requirements  and  defend  against  IED  attacks.  Additionally,  inconsistency  within 
the  proposal  evaluation  process  could  be  disastrous  for  an  organization  that  fails  to 
adequately  justify  their  reasons  for  making  important  decisions.  It  is  clear  that  these 
weaknesses  are  at  the  front  of  the  GAO’s  motivation  for  investigating  JIEDDO ’s  current 
lines  of  operations,  among  which  include  proposal  evaluation  for  selection  or  rejection  in 
the  earliest  stages  of  the  acquisition  process. 

The  importance  of  utilizing  a  valid  value  model  that  adequately  justifies  the 
proposal  evaluation  process  is  at  the  core  of  this  research.  N-way  sensitivity  analysis  is 
not  unique  to  the  JIEDDO  decision  model  because  most  decision  problems  contain  more 
than  two  values.  Even  the  simplest  day-to-day  decisions  require  an  assessment  of  a 
number  of  different  values.  N-way  sensitivity  analysis  is  required  to  provide  a 
comprehensive  evaluation  of  the  alternatives  for  a  decision  problem  prior  to 
implementation. 

I.C.  Research  Scope 

This  thesis  will  evaluate  the  validity  of  the  proposed  JIEDDO  Proposal  Value 
Model.  As  such,  this  research  addresses  the  following  questions:  Does  the  value  model 
adequately  reflect  JIEDDO ’s  decision  process?  Secondly,  given  the  dynamic 
environment  of  the  current  war,  how  confident  can  we  be  in  the  model ’s  ability  to 
continually  and  effectively  screen  proposals  based  on  JIEDDO ’s  current  values?  To 
address  the  first  question,  the  author  will  utilize  multivariate  techniques,  specifically 
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Discriminant  Analysis,  to  investigate  JIEDDO’s  ability  to  make  consistent  decisions 
within  their  proposal  evaluation  process.  Once  it  has  been  detennined  that  the  JIEDDO 
Proposal  Value  Model  effectively  screens  proposals  in  nearly  the  same  manner  as  that  of 
a  panel  of  decision  makers,  it  is  possible  to  proceed  with  the  second  question. 

In  order  to  address  the  model’s  robustness,  the  author  will  investigate  the 
weighted  values  that  the  model  utilizes  to  evaluate  a  set  of  given  proposals.  By 
consolidating  and  applying  n-way  sensitivity  analysis  techniques,  specifically  in  the  areas 
of  math  programming  and  Multivariate  Analysis,  the  author  will  propose  a  consistent 
sensitivity  analysis  image  profiling  technique. 

I.D.  Assumptions 

Value  focused  thinking  models  are  carefully  developed  by  decision  analysts  who 
work  with  the  decision  maker  to  encapsulate  adequately  the  values  for  the  decision  under 
consideration.  This  research  assumes  that  the  creators  of  the  original  JIEDDO  Proposal 
value  model  worked  with  the  appropriate  decision  maker  and  subject  matter  experts  to 
identify  all  of  the  values  pertaining  to  the  decision  problem  at  hand.  Furthermore,  the 
assumption  is  that  all  of  the  values  for  the  decision  have  been  captured.  Lastly,  this 
research  assumes  the  appropriate  application  of  the  value  focused  thinking  methodology 
(VFT)  requirements  to  include  small  size,  operability,  mutual  exclusivity,  preferential 
independence,  and  collectively  exhaustive  for  the  JIEDDO  Proposal  Value  Model. 

I.E.  Thesis  Organization 

The  remainder  of  this  thesis  contains  four  chapters  organized  in  the  following 
manner:  Chapter  2  consists  of  a  thorough  literature  review  of  DA,  VFT  methodology, 
value  hierarchy  construction,  weighting,  sensitivity  analysis,  as  well  as  field  applications. 


Chapter  3  of  this  document  presents  a  discriminate  analysis  model  validation  technique. 
Additionally,  the  chapter  outlines  and  presents  a  new  n-way  sensitivity  profiling 
technique.  Chapter  4  consists  of  the  results  and  analysis  that  comprise  the 
aforementioned  discriminant  analysis  techniques  and  the  in-depth  n-way  sensitivity 
analysis  as  it  pertains  to  the  JIEDDO  proposal  solicitation  process.  Lastly,  chapter  5 
discusses  relevant  conclusions  and  explores  opportunities  for  future  research. 
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II.  Literature  Review 


II.A  Introduction 

The  purpose  of  the  literature  review  is  to  conduct  an  extensive  study  on  the 
various  components  of  DA  to  include  VFT,  value  hierarchy  construction,  weighting, 
sensitivity  analysis  as  well  as  the  details  surrounding  the  JIEDDO  proposal  selection 
decision  problem.  A  thorough  understanding  of  the  JIEDDO  decision  model  is  rooted  in 
DA  methodology.  The  JIEDDO  value  model  utilizes  an  additive  value  function  to  model 
the  importance  of  a  series  of  values  sought  in  the  submitted  proposals.  Each  of  the 
identified  values  is  assigned  a  weight  that  captures  its  relative  importance  for  the  decision 
problem.  An  evaluation  of  the  sensitivity  of  the  weights  provides  insight  into  the 
robustness  of  the  model  itself. 

II.B  Decision  Analysis 

Decision  Analysis  originated  during  the  1950s  when  Robert  Schlaifer  introduced 
some  of  the  analytical  techniques  in  his  book  Analysis  of  Decisions  under  Uncertainty 
(Skinner,  1999,  17).  Corporate  decision  makers  in  our  current  society  utilize  DA  because 
they  recognize  its  usefulness  in  the  decision  making  process  (Skinner,  2001,  9). 

According  to  Clemen,  before  we  can  begin  to  apply  any  methodology  to  a  specific 
decision  problem,  the  analyst  must  first  identify  a  decision  maker  who  is  appropriate  for 
the  decision  and  possesses  the  proper  authority  thereof  (2001).  After  identifying  the  DM, 
it  is  possible  to  work  with  them  to  determine  the  values  and  objectives  that  relate  to  the 
decision  in  question  (Clemen,  2001,  21).  Clemen  clarifies  the  distinction  between 
objectives  and  value,  stating  “An  objective  is  a  specific  thing  that  you  want  to 
achieve. .  .An  individual’s  objectives  taken  together  make  up  his  or  her  values”  (2001, 


10 


22).  While  most  people  subconsciously  make  decisions  for  their  day-to-day  life  based  on 
their  values,  we  cannot  guarantee  the  selection  of  the  best  alternative  unless  we 
conscientiously  apply  a  specific  methodology  that  is  a  repeatable. 

Based  on  the  decision  problem,  decision  analysts  apply  one  of  two  different 
methodologies  to  decision  processes  where  single  dimensional  value  functions  are 
involved— Value  Focused  Thinking  (VFT)  or  Alternative  Focused  Thinking  (AFT). 
Keeney  explicitly  differentiated  between  VFT  and  AFT  in  Value-Focused  Thinking:  A 
Path  to  Creative  Decisionmaking,  “Value-focused  thinking  involves  starting  at  the  best 
and  working  to  make  it  a  reality.  Alternative-focused  thinking  is  starting  with  what  is 
readily  available  and  taking  the  best  of  the  lot”  (1992,  6).  As  a  result,  VFT  first 
approaches  the  decision  problem  by  identifying  all  of  the  values  relating  to  the  decision. 
The  idea  in  VFT  is  to  vocalize  the  desire  for  an  alternative  to  possess  a  certain  set  of 
values,  and  investigate  the  feasibility  of  producing  one.  AFT  on  the  other  hand,  uses  the 
available  alternatives  as  the  starting  point  for  the  decision  process.  AFT  will  identify  the 
best  alternative  out  of  a  list  of  provided  alternatives. 

II.C  Value  Focused  Thinking 

The  appeal  to  VFT  is  its  ability  to  challenge  the  DM  to  produce  a  highly  desirable 
alternative.  According  to  Keeney  in  Value  Focused  Thinking:  A  Path  to  Creative 
Decisionmaking,  the  process  involves  “two  activities”:  “Decide  what  you  want  and  then 
figuring  out  how  to  get”  (1992,  p.  4).  After  properly  identifying  the  DM’s  values  and 
then  arranging  them  using  affinity  grouping  or  some  other  technique,  it  is  possible  to 
determine  the  means  by  which  to  measure  the  values  (Knighton,  2008).  Next,  the  analyst 
applies  the  weights  in  order  to  score  the  alternatives  based  on  an  additive  value  function 
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created  using  the  weighted  sum  of  the  DM’s  previously  identified  values.  The  proposed 
alternatives  then  receive  a  score  based  on  the  characteristics  they  possess  in  relation  to 
the  DM’s  values.  Ideally,  the  generated  alternatives  possess  qualities  or  characteristics 
that  fonn  the  most  appealing  solution  (Keeney,  1992). 

II.D  Value  Hierarchy  Creation 

Upon  deciding  to  apply  VFT  methodology  to  the  decision  problem,  it  is  important 

to  construct  a  value  hierarchy  that  adequately  models  the  problem  under  consideration. 

According  to  Keeney,  four  steps  describe  value  hierarchy  creation: 

The  first  is  to  work  with  a  decision  maker  to  determine  the  set  of  objectives  that 
are  appropriate  for  the  decision  under  consideration.  Second,  the  analysts  define 
attributes  that  accurately  measure  how  well  the  objectives  are  met.  Third,  a 
reasonable  structure  or  hierarchy  combines  the  varying  attributes  in  an  orderly 
manner.  Lastly,  the  hierarchy  is  verified  and  its  reasonability  is  examined  to 
detennine  compatibility  with  the  situation  at  hand  (Keeney,  1992,  p.  131). 

The  analyst  works  with  the  DM  to  extract  all  of  the  values  for  the  decision.  Grouping  the 

values  allows  us  to  consolidate  similar  values  and  create  tiers  for  the  respective 

subcategories  for  each  of  the  values. 

In  Strategic  Decision  Making,  Kirkwood  defines  a  value  hierarchy  as  including 
evaluation  considerations,  objectives,  and  evaluation  measures  (Kirkwood,  1997,  15). 
Furthennore,  he  outlines  five  properties  necessary  for  creating  a  value  hierarchy: 
collectively  exhaustiveness,  mutually  exclusivity,  decomposability,  operability,  and  small 
size.  The  requirement  that  the  model  be  collectively  exhaustive  ensures  that  it  contains 
all  relative  infonnation  for  the  decision  problem.  Mutual  exclusivity  ensures  that  no  two 
evaluation  considerations  overlap  as  to  avoid  “double  counting”  (Kirkwood,  1997,  17). 

In  addition  to  the  value  hierarchy  being  collectively  exhaustive  and  mutually  exclusive,  it 
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is  necessary  to  preserve  decomposability,  also  known  as  preferential  independence.  One 
evaluation  measure  may  be  mutually  exclusive  from  another;  however,  they  may  not  be 
preferentially  independent  of  one  another.  Kirkwood  proposes  the  following  example  to 
reiterate  this  point: 

Suppose  that  a  job  seeker  has  as  an  evaluation  consideration  economic  issues,  and 
has... evaluation  considerations... salary,  pension  benefits,  and  medical  coverage. 
Notice  that  these  are  non-redundant  issues,  but  they  may  not  be  decomposable.  If 
there  are  very  good  pension  benefits,  then  the  value  of  an  additional  $5,000  in 
salary  may  be  less  than  if  the  pension  plan  is  poor  and  the  job  seeker  will  need  to 
provide  for  his  or  her  retirement  out  of  salary  (Kirkwood,  1997,  17). 

Thus,  preferential  independence  serves  a  significant  contribution  to  the  architecture  of  the 

value  hierarchy. 

Next,  the  operability  of  the  value  hierarchy  describes  the  intended  audience’s 
ability  to  understand  the  model.  By  preserving  operability,  the  analyst  ensures  the 
transferability  of  the  model  details  to  the  decision  maker  or  another  key  figure  (Knighton, 
2008).  This  individual  is  then  able  to  relay  the  details  of  the  decision  problem  and  value 
hierarchy  construction  over  to  the  public  whose  acceptance  is  crucial  to  the 
implementation  of  the  decision. 

Lastly,  according  to  Kirkwood,  the  size  of  the  value  hierarchy  is  of  key 
importance.  It  is  easier  to  communicate  a  small  value  hierarchy  to  a  variety  of  audiences, 
than  it  is  to  communicate  a  large  hierarchy  (Kirkwood,  1997,  18).  Furthermore,  creating 
a  small  value  model  often  makes  it  easier  to  conduct  analysis  and  interpret  results. 

II.E  Weighting 

Once  the  value  hierarchy  is  constructed,  Clemen  and  Reilly  suggest  that  a  trade 
off  is  made  between  varying  objectives.  They  introduce  weights  to  detennine  the  exact 
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trade  off.  Furthermore,  they  require  that  the  weights  appropriately  reflect  “the  relative 
value”  of  going  from  best  to  worst  on  each  scale  for  a  particular  attribute  (2001).  Thus, 
weights  reflect  the  DM’s  preference. 

In  Strategic  Decision  Making,  Kirkwood  provides  a  detailed  discussion  for 
detennining  the  weights  for  a  given  value  model.  The  value  function  for  a  given  decision 
problem  with  n-different  evaluation  measures  is  defined  in  Equation  1  below. 

Equation  1  (Kirkwood,  61) 

v(,Xa,  ,  Xc,  ■  ■  ■  Xn )  wa va (Xa )  +  Wb  (Xb )  +  wc vc (,XC )  T  •••  +  wn  vn (Xn ) 

Xt  =  evaluation  measure  for  the  7th  value 

w;  =  weight  on  the  7th  evaluation  measure 

Vi(Xt)  =  single  dimensional  value  functions  over  each  of  the  i  measures 

7  =  a  particular  value 

The  importance  of  a  given  evaluation  measure  is  denoted  by  the  assigned  weight,  w;. 
Those  values  that  possess  a  relatively  high  weight  signify  a  greater  level  of  importance 
for  the  decision  problem.  As  a  result,  selected  alternatives  typically  have  highly  desired 
characteristics.  However,  the  total  score  detennination  is  dependent  upon  the  sum  of  the 
weighted  single  dimensional  value  functions. 

Kirkwood  describes  the  technique  of  swing  weighting  as  a  tool  for  adequately 
detennining  weights.  It  is  necessary  to  order  the  increments  in  value  by  increasing  or 
“swinging”  each  of  the  evaluation  measures  from  least  desirable  to  most  desirable 
(Kirkwood,  1997).  After  organizing  the  values  incrementally  in  order  of  importance, 
quantitatively  scale  each  of  the  value  increments  as  a  multiple  of  the  smallest  value 
increment  (Kirkwood,  1997).  Once  this  value  referencing  determination  is  complete  each 
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of  the  scaled  weights  are  summed  to  1  and  the  weight  for  the  least  valuable  measure  is 
detennined  (Kirkwood,  1997). 

This  swing  weighting  technique  is  examined  in  an  example  for  the  characteristics 
sought  in  a  car  prior  to  purchasing.  We  may  consider  the  following:  safety  rating,  color, 
size.  Now,  ordering  these  values  incrementally  from  least  preferred  to  most  preferred,  we 
have:  color,  size,  safety  rating.  Color  receives  a  weight  of  k.  Now,  the  decision  maker 
detennines  how  much  more  they  prefer  having  the  vehicle  size  of  choice  over  that  of 
color  choice.  For  illustrative  purposes,  the  decision  maker  says  2*k.  Next,  we  detennine 
how  much  more  important  is  the  value  of  safety  rating  over  that  of  car  color.  The 
decision  maker  responds  4*k.  Following  Kirkwood’s  procedure  for  determining  the 
single  dimensional  value  function,  we  have  the  following  calculations: 

k  +  2k  +  4k  =  1 
7k  =  1 

Solving  the  algebraic  equation,  k  =  1/7 

Thus,  the  weights  for  each  of  the  values  color,  size,  and  safety  rating  are  1/7,  2/7,  4/7 
respectively. 

II.F  Sensitivity  Analysis 

The  question  of  the  sensitivity  of  a  decision  problem  under  given  pre-determined 
weights  refers  to  the  variables  that  really  make  a  difference  in  terms  of  the  decision  under 
consideration  (Clemen  &  Reilly,  2001).  Clemen  and  Reilly  present  methods  for 
conducting  one-way  and  two-way  sensitivity  analysis  as  to  see  how  fluctuating  one 
weight  while  holding  the  remaining  weights  proportionately  constant  (similarly,  altering 
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two  weights,  while  holding  the  remaining  weights  proportionately  constant)  affects  the 
rank  order  of  alternatives  for  a  given  decision  problem  (Clemen  &  Reilly,  2001). 

One-way  sensitivity  analysis  allows  us  to  observe  how  sensitive  the  decision 
problem  is  by  looking  for  rank  changes  among  alternatives  as  a  single  weight  fluctuates 
between  zero  and  one  (Clemen  &  Reilly,  2008).  If  a  rank  change  occurs  from  adjusting  a 
weight  by  a  small  amount,  then  the  decision  problem  is  sensitive  for  that  particular 
weight.  To  illustrate  this  further,  consider  one-way  sensitivity  analysis  previously 
conducted  by  Dawley,  Marentette,  and  Long  on  Gap  Impact. 


Figure  1:  Sensitivity  of  Gap  Impact  (Dawley,  Marentette,  &  Long,  2008) 

The  original  global  weight  for  Gap  Impact,  indicated  by  the  black  line  above,  is 
0. 176.  The  analyst  observes  the  sensitivity  of  ten  distinct  proposals  to  varying  the  weight 
on  Gap  Impact  between  zero  and  one.  As  the  weight  on  Gap  Impact  decreases,  the  values 
for  Proposals  W,  X,  and  P*  increase  while  the  values  for  Proposals  R,  D*,  T,  and  C* 
decrease.  Thus,  rank  changes  occur  resulting  from  adjustments  of  the  weight  for  Gap 
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Impact.  The  same  methodology  applies  when  considering  the  case  of  varying  two 
weights  simultaneously  and  holding  the  remaining  weights  proportionately  constant. 

In  an  article  of  the  application  of  Response  Surface  Methodology  (RSM)  as  a 
sensitivity  analysis  tool  in  DA,  Bauer,  Parnell,  and  Meyers  acknowledge  the  benefits  of 
perfonning  one-way  and  two-way  sensitivity  analyses  as  a  “screening  tool”  for  many 
problems.  However,  they  suggest  that  a  lack  of  higher  order  sensitivity  analysis  to 
include  n  simultaneous  weight  changes  has  the  potential  of  excluding  a  large  portion  of 
information  that  is  contained  in  the  model  (Bauer,  Parnell,  &  Meyers,  1999).  They 
present  RSM  techniques  using  the  output  of  the  model  as  the  “response”  and  its  input 
variables  in  order  to  improve  the  decision  model  (Bauer,  Parnell,  &  Meyers,  1999).  The 
analysts  investigate  the  “numeric  perturbation  range”  for  each  variable  to  determine  the 
region  of  operability  (Bauer,  Parnell,  &  Meyers,  1999,  165).  Through  the  use  of  a  “design 
matrix”,  where  each  row  represents  a  set  of  experimental  conditions,  it  is  then  possible  to 
observe  n  different  realizations  of  the  model  of  interest  (Bauer,  Parnell,  &  Meyers,  1999, 
165).  According  to  Bauer  et  ah,  the  requirement  for  using  a  Response  Surface  Paradigm 
is  to  identify  a  response  function  of  the  form  y  =  f(x)  (1999,  167).  Recognizing  that 
multi-attribute  preference  theory  and  multi-attribute  utility  theory  allows  us  to  write  the 
value  function  in  this  form,  they  conclude  that  RSM  is  appropriately  used  to  detennine 
the  sensitivity  of  the  decision  problem  for  n  varying  weights. 

Rios  Insua  and  French  provide  a  distance  based  methodology  to  the  sensitivity 
analysis  problem  for  discrete  multi-objective  decision-making.  Through  using  math 
programming  techniques,  they  compare  alternative  a j  with  a  “possible  competitor”  a* 
(where  a*  is  the  optimal  alternative  for  the  current  decision  problem)  (Rios  Insua  & 


17 


French,  1989,  181).  The  authors  use  this  technique  in  order  to  determine  by  how  much 

weight  w  can  be  varied  before  aj  outranks  a*.  Thus,  they  suggest  solving  the 

minimization  problem: 

Equation  2  (Rios  Insua  &  French,  1989,  181) 

min  d(w,o>) 
weS 

s.t.  *Fj(wj)-  T*  (w)=0 
where: 

d:  distance  metric 
w:  new  weights 
c o :  original  weights 
S:  constraints  on  w 

T**  (w):  score  provided  for  the  optimal  alternative  at  new  weights  w 
'Fj(wj):  score  provided  for  alternative)  at  new  set  of  weights  Wj 

This  provides  insight  as  the  minimization  problem  seeks  to  determine  Wj  such  that  aj  and 

a*  are  indifferent  (Rios  Insua  &  French,  1989,  181).  They  take  this  idea  a  step  further  by 

analyzing  several  distances  (dj1,...dJn)  to  determine  the  efficient  alternatives  of  the 

problem  of  minimizing  these  distances  accordingly.  Thus,  they  allude  to  the  use  of 

varying  distances  in  n-ways  to  determine  the  sensitivity  of  the  weights  for  a  given  set  of 

alternatives. 

Hughes  and  Hughes  apply  n-way  sensitivity  analysis  to  the  field  of  medicine  in  N- 
Way  Sensitivity  Analysis:  A  Verification  and  Stability-assessment  Technique  Completely 
Subjective  Decision  Analysis  Trees,  published  in  Medical  Decision  Making  in  1990.  The 
authors  present  a  case  for  using  the  absolute  mean  difference  estimation  of  variance  in 
lieu  of  the  standard  deviation  calculations  of  variance  for  a  study  looking  at  the  extent  to 
which  nurses  make  “internally  consistent  and  mathematically  logical  decisions  as  well  as 
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to  construct  demographic,  experimental,  and  educational  profile  of  the  consistent  clinical 
decision  makers”  (Hughes  &  Hughes,  1990,  69).  While  they  acknowledge  the  case  for 
standard  deviation  calculations  of  variance  in  some  instances  of  n-way  sensitivity 
analysis,  they  acknowledge  it’s  inappropriateness  in  the  context  of  their  decision 
problem.  This  justification  is  based  on  a  very  small  number  of  parameters  (4  of  83  to  be 
exact)  failing  to  exhibit  normality  according  to  the  Kolmogrov-Smirnov  test  (Hughes  & 
Hughes,  1990,71-72). 

In  a  published  article,  Sensitivity  Analysis  of  Additive  Multiattribute  Value 
Models,  Barron  and  Schmidt  describe  the  least  squares  method  for  determining  the 
sensitivity  of  the  decision  problem  to  varying  weights.  They  compute  the  weights 
w={wi  ,W2,. .  .wn}  to  determine  the  value  of  the  alternative  vn  that  exceeds  that  of  the 
current  optimal  alternative  v*  by  only  a  small  amount  (Barron  &  Schmidt,  1988,  123). 
Thus,  they  intentionally  choose  weights  very  near  the  weights  for  vn  (1988).  Barron  and 
Schmidt  compute  the  L2-norm  calculating  the  squared  deviation  of  the  weights  (1988, 
123).  By  applying  this  least  squares  method,  they  conclude  that  an  alternative  that 
surpasses  that  of  the  value  for  the  current  optimal  solution  for  a  given  set  of  weight  very 
near  that  of  the  optimal  proves  to  be  very  sensitive  for  the  decision  problem  (Barron  & 
Schmidt,  1988,  123). 

Ringuest  provides  an  extension  of  current  L2  metric  sensitivity  analysis 
methodology  by  considering  solutions  that  minimize  the  Li  and  Loo-metrics  subject  to  a 
set  of  linear  constraints  (1996,  563).  He  suggests  the  Lp-metric  as  the  generalized  fonn 
of  Barron  and  Schmidt’s  least  squares  procedure  allowing  for  a  P  effect  on  the  “relative 
contribution  of  individual  deviations”  (1996).  Thus  for  large  values  of  P  a  larger 
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contribution  of  individual  deviations  is  considered,  while  smaller  values  of  P  have  a  less 
significant  contribution  to  the  individual  deviations  (Ringuest,  1996).  Furthennore,  such 
a  variation  of  P,  allows  for  a  maximum  change  in  any  one  multi-attribute  value  function 
weight  to  be  achieved  when  P=  oo  (Ringuest,  1996). 

In  2008,  Marks  researched  the  affects  of  n-way  sensitivity  analysis  on  various 
courses  of  action  for  the  Iraq  war.  His  research  pertains  to  the  development  of  a  Value 
Focused  Thinking  model  to  score  a  series  of  courses  of  action  options.  Marks  elicits  14 
evaluation  measures  from  his  team  of  SMEs  to  include:  the  Percentage  of  Coalition  Lost, 
the  Number  of  Insurgents  Crossing,  Ethno-Sectarian  Violence,  Non-sectarian  Violent 
Death  Rate,  Average  Hours/Day,  Water  Available  per  Person  per  Day,  Fuel  Available  per 
Person  per  Day,  Tons  of  Supplies  Needed,  Estimated  Number  of  Members,  Willingness, 
and  Addition  to  the  Number  of  Units  at  Levels  One  through  Four  (each  unit  level  has  a 
separate  single  dimensional  value  function)  (2008).  Upon  eliciting  these  values,  Marks 
worked  with  the  team  to  extract  weights  that  would  appropriately  reflect  that  of  a 
Combatant  Commander  who  faces  selecting  one  from  a  series  of  courses  of  action.  The 
weighted  value  function  or  additive  value  function  allowed  the  team  to  score  a  series  of 
10  different  courses  of  actions  based  on  the  value  model.  From  these  scores,  Marks 
developed  a  ranking  of  most  preferred  course  of  action  to  least  preferred  course  of  action. 

While  the  ranking  of  courses  of  action  proved  insightful  based  on  the  values 
defined  in  the  model,  questions  remain  as  to  the  level  of  confidence  a  Combatant 
Commander  can  have  in  utilizing  a  top  ranked  alternative.  Marks  acknowledges  that  the 
dangers  of  selecting  the  wrong  course  of  action  can  have  significant  consequences. 
American  troops  could  be  put  unnecessarily  in  harm’s  way,  and  civilian  and  military  lives 
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could  be  lost  if  a  course  of  action  is  implemented  that  did  not  consider  all  aspects  of  the 
situation  at  hand  (Marks,  2008). 

In  order  to  determine  the  level  of  confidence  that  lies  in  the  model  itself,  Marks 
recognizes  the  need  to  examine  the  weights  assigned  to  each  of  the  identified  values. 

“Are  the  weights  really  a  reflection  of  the  decision  maker’s  preferences”  (Marks,  2008)? 
Recognizing  the  dynamic  environment  commander’s  face  during  the  current  war,  Marks 
seeks  to  detennine  if  small  adjustments  of  the  weights  will  significantly  affect  the  rank 
order  of  the  suggested  courses  of  action  (2008). 

To  address  these  questions,  Marks  applies  math  programming  techniques  to 
conduct  n-way  sensitivity  analysis  on  the  weights  for  the  Course  of  Action  Value  Model. 
Particularly,  he  compared  the  results  produced  from  optimizing  the  weight  change  for  the 
Li,  L2,  Loo  norms,  sum  of  squares,  as  well  as  a  percent  change  metric  (2008).  He 
believes  such  a  comparison  will  shed  light  on  the  sensitivity  of  the  weights  themselves 
and  ultimately  provide  an  answer  to  the  questions  at  hand. 

He  evaluates  the  sensitivity  of  10  different  alternatives,  including  Self-sustained 
Agriculture,  Training  Indigenous  Security  Institutions,  Expelling  A1  Qaeda-Iraq, 
Instituting  a  Military  Draft  in  Iraq,  Partial  Coalition  Withdrawal  from  Iraq  and  Full 
Coalition  Withdrawal  from  Iraq.  Marks  seeks  to  reveal  the  effects  of  higher  order  weight 
variations  on  these  courses  of  action  (2008).  In  comparing  the  effects  of  n-way  weight 
changes,  specifically  the  aforementioned  14  values,  Marks  determines  that  half  of  the 
alternatives  are  sensitive  (2008).  As  a  result  of  the  generality  that  is  gained  from  such 
small  weight  changes,  he  recommends  using  the  1  -norm  as  a  primary  means  of 
determining  the  overall  sensitivity  of  the  weights  to  conduct  sensitivity  analysis  (Marks 
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2008).  However,  he  does  not  discount  the  use  of  any  one  of  the  remaining  four  math 
programming  options  as  an  invalid  sensitivity  analysis  technique. 

II.G  JIEDDO 

The  Department  of  Defense  formally  recognized  a  growing  need  for  a  joint  C-IED 
collaborated  effort  in  February  2006  by  establishing  JIEDDO.  They  declared  their 
primary  focus  as  “reducing  or  eliminating  the  effects  of  all  fonns  of  IEDs  used  against 
U.S.  and  Coalition  Forces,  including  policy,  resourcing  materiel,  technology,  training, 
and  operations,  information,  intelligence,  assessment  and  research”  (Defense,  2006). 
JIEDDO ’s  ability  to  counter  IEDs  effectively  is  largely  a  reflection  of  their  ability  to 
engage  the  public  to  aid  in  the  response  to  research  and  develop  C-IED  programs. 

In  order  to  aid  the  decision  process,  in  2008  Dawley,  Marentette,  and  Long 
developed  a  value  model  to  define  the  JIEDDO  initiative  solicitation  process. 

Recognizing  that  there  is  currently  no  repeatable  framework  in  place  to  assist  JIEDDO  in 
selecting  proposals  for  funding  C-IED  projects,  the  authors  sought  to  provide  a 
systematic  methodology  to  the  process.  Figure  2  shows  the  complete  value  hierarchy  and 
the  associated  weights  for  each  value. 
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Figure  2:  Complete  JIEDDO  Value  Hierarchy  (Dawley,  Marentette,  &  Long,  2008) 

The  “Potential  to  Defeat  IED”  contains  three  first  tier  values:  Needed  Capability, 
Operational  Performance,  and  Usability  (Dawley,  Marentette,  &  Long,  2008).  The 
architects  of  the  model  used  SMEs  to  extract  the  weights  for  each  tier  of  the  model  to 
encapsulate  the  appropriate  importance  of  each  evaluation  measure  adequately. 

The  information  used  to  score  the  proposals  is  extracted  from  white  papers  and 
quad  charts  provided  by  the  applicant  pool  at  large.  The  proposal  information  is 
extracted  from  the  current  proposal  solicitation  database  by  means  of  the  Broad  Area 
Announcement  Information  Delivery  System  (BIDS)  (Dawley,  Marentette,  &  Long, 
2008).  The  information  collected  via  BIDS  is  traditionally  used  by  a  panel  of  subject 
matter  experts  to  determine  whether  the  proposed  initiative  will  adequately  meet  the 
JIEDDO ’s  C-IED  objectives.  Through  utilizing  DA  techniques,  the  models  creators 


23 


present  a  case  for  producing  a  value  model  that  allows  JIEDDO  to  screen  proposals 
rapidly  to  determine  if  they  are  appropriate  to  enter  the  initiative  evaluation  stage. 

While  the  authors  have  confidence  in  the  value  model  and  its  usefulness  to 
JIEDDO,  they  recognize  that  advancement  is  necessary  in  the  way  of  n-way  sensitivity 
analysis.  The  team  conducts  extensive  one-way  sensitivity  analysis  to  investigate  the 
rank  order  of  the  proposal  sets.  However,  the  overall  scores  for  each  of  the  proposed 
initiatives  are  very  close  (Dawley,  Marentette,  &  Long,  2008).  This  poses  some  concern 
into  the  sensitivity  of  the  decision  under  such  small  weight  changes  (Dawley,  Marentette, 
&  Long,  2008).  Furthermore,  the  authors  recognize  that  they  conducted  sensitivity 
analysis  on  a  discrete  set  of  alternatives.  The  task  of  analyzing  the  sensitivity  of  the 
selection  of  alternatives  for  a  continuous  evaluation  solicitation  process  proves  to  be  very 
challenging  (Dawley,  Marentette,  &  Long,  2008). 

The  sensitivity  analysis  falls  short  beyond  one-way  sensitivity  analysis  and  the 
implications  of  altering  even  two  of  the  weights  are  limited  for  this  study.  Relying  on  the 
SME  to  pinpoint  all  of  the  weights  exactly  may  prove  troublesome  for  JIEDDO  in  the 
future  without  the  demonstration  of  extensive  model  validation. 

II.H  Summary 

This  chapter  presents  research  from  literature  in  the  areas  of  DA  to  include  VFT, 
value  hierarchy  construction,  weighting,  and  sensitivity  analysis.  Among  the  themes  of 
importance  is  the  comprehensiveness  of  the  value  model  itself,  the  appropriateness  of  the 
weights,  as  well  as  the  sensitivity  analysis  for  the  respective  weights.  Of  particular 
interest  to  the  analyst  team  is  the  decision  maker’s  confidence  in  the  model  to  identify  the 
decision  at  hand.  Much  of  the  DM’s  ability  to  proceed  with  a  particular  alternative  for 
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the  decision  replies  upon  their  belief  that  they  are  making  the  correct  one  for  their 
particular  organization.  This  alternative  selection  process  proves  to  be  of  key  importance 
to  the  leadership  at  JIEDDO  as  they  detennine  which  proposals  they  will  select  for 
funding.  The  next  chapter  expands  upon  the  current  methodology  for  robust  sensitivity 
analysis  to  include  a  particular  application  to  the  JIEDDO  proposal  selection  value 
model. 


25 


III.  Methodology 


III.A.  Introduction 

The  purpose  of  this  chapter  is  to  describe  the  methods  used  to  evaluate  the 
usefulness  and  robustness  of  the  JIEDDO  Proposal  Value  Model.  We  recognize  that  the 
dynamic  environment  of  the  current  War  on  Terror  justifiably  motivates  us  to  deliver  the 
most  effective  counter- IED  initiatives  to  the  warfighter.  Thus,  the  methods  developed  in 
this  chapter  serve  a  dual  purpose.  First,  this  research  will  utilize  multivariate  techniques 
to  answer  the  following  question,  “Does  the  model  replicate  JIEDDO ’s  decision 
process?”  If  the  model  does  in  fact  adequately  replicate  JIEDDO’s  decision  process,  then 
it  would  serve  as  a  rapid  screening  tool  for  the  pre-proposal  investigation  process. 

Second,  given  the  potential  for  using  it  as  a  rapid  screening  tool,  how  confident  can  we  be 
in  the  model’s  results?  Provided  that  the  model  supplies  a  ranking  of  proposals  from  best 
case  to  worst  case,  we  want  a  level  of  assurance  that  the  rank  order  is  robust.  This  is 
detennined  by  observing  the  affect  on  rank  order  under  simultaneous  weight  changes 
within  the  model  itself. 

III.B  Discriminant  Analysis 

Discriminant  Analysis  is  a  technique  that  is  used  to  “classify  individuals  or 
objects  into  mutually  exclusive  and  collectively  exhaustive  groups  based  on  an  observed 
set  of  independent  variables”  (Bauer,  Parnell,  &  Meyers,  1999).  In  order  to  apply  this 
technique,  the  analyst  splits  the  original  dataset  into  a  training  sample  and  a  validation 
sample.  For  data  sets  with  a  large  number  of  samples,  the  training  sample  is  comprised 
of  2/3  of  the  observations  from  the  original  data  while  the  validation  sample  comprises 
the  remaining  1/3.  The  classification  of  a  particular  object  into  a  group  is  determined  a 
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priori  by  means  of  a  discriminant  function  defined  using  the  training  data  (Bauer  K.  W., 
2008).  Each  classification  group  possesses  a  specific  discriminant  function  that  describes 
each  group  of  interest.  Next,  the  approach  involves  attaching  a  scalar  score  to  each 
object  in  the  validation  data  for  each  of  the  defined  groups.  The  analyst  calculates  the 
scores  by  taking  a  linear  combination  of  the  object’s  attributes.  The  object  becomes  a 
member  of  the  group  whose  discriminant  function  produces  the  highest  numerical  value 
when  given  the  object’s  specific  input  values.  The  discriminant  function  fonnulation 
utilized  is  of  the  form  given  in  Equation  3  below. 

Equation  3  (Bauer  K.  W.,  2008) 

where: 

S  j :  the  covariance  matrix  for  the  /th  population  (i.e.  accept  population,  reject 
population) 

X0 :  a  specific  observation  that  is  seeking  to  be  classified 

X. :  the  mean  of  the  observations  that  comprise  the  /lh  population 

R :  the  previous  population  percentage  of  the  /'th  group 

In  order  to  categorize  the  proposals  appropriately  into  an  accept  or  reject  group, 
the  analyst  must  utilize  the  infonnation  collected  for  each  of  the  proposals.  The  13 
values  depicted  in  the  JIEDDO  “Potential  to  Defeat”  hierarchy  shown  in  Figure  2  capture 
significant  pieces  of  information  for  the  JIEDDO  decision  problem.  The  model’s 
architects  provided  a  brief  summary  of  these  13  values  in  Table  1. 
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Table  1:  JIEDDO  Values  (Dawley,  Marentette,  &  Long,  2008) 


Variable 

Definition 

T enets  Impacted 

Answers  the  question,  “Does  this  proposal  impact  one  or  many 
tenets?”  This  value  allows  the  hierarchy  to  capture  the  synergistic 
value  of  a  solution  that  impacts  more  than  one  tenet. 

Gap  Impact 

JIEDDO  establishes  prioritized  capability  gaps  on  a  periodic  basis 
with  input  from  the  Combatant  Commands.  Thus,  the  value  of  a 
proposal  is  directly  related  to  the  priority  of  the  gap  which  it 
targets. 

Classification 

How  easily  can  this  solution  be  shared  among  stakeholders  within 
and  outside  of  DoD? 

Time  to  Counter 

How  long  will  it  take  for  the  enemy  to  develop  a  counter-measure 
for  the  system? 

Technical  Performance 

The  predicted  performance  of  a  system  while  executing  its  intended 
mission. 

Suitability 

How  well  the  system  will  perform  in  its  intended  environment. 

Interoperability 

The  degree  to  which  a  system  fits  into  the  existing  network 
architecture,  whether  it  can  exchange  data  with  supporting  and 
supported  systems,  and/or  whether  it  can  perform  its  task  without 
negatively  impacting  friendly  assets. 

Technical  Risk 

JIEDDO  is  a  risk-tolerant  organization.  They  are  willing  to  accept 
technology  risk  if  it  is  outweighed  by  other  benefits.  However,  a 
mature  technology  will  provide  more  value  than  an  unproven 
technology  for  the  same  performance. 

Fielding  Timeline 

How  soon  the  solution  can  be  fielded.  If  a  solution  can’t  be  fielded 
in  a  timely  manner  it  becomes  much  less  relevant  to  JIEDDO — 
only  10%  of  their  budget  is  spent  on  proposals  with  a  timeline  of  3 
years  or  longer.  “Fielded”  is  defined  as  when  the  first  article  of  a 
system  is  delivered. 

Operations  Burden 

The  degree  to  which  the  system  will  impact  the  capacity  of  its  host 
environment,  e.g.  bandwidth  required  by  a  collaborative  software 
system,  weight  for  a  vehicle  or  soldier  mounted  system,  or  rack 
space  required  for  server  enabled  analysis  system. 

Work  Load 

This  value  captures  the  time  requirements  that  the  solution  places 
on  the  user  in  an  operational  environment  to  ensure  that  the  system 
continues  to  operate  as  expected. 

Training  Time 

How  long  does  it  take  to  train  the  average  user  on  the  solution? 

Appendix  A  contains  the  detailed  proposal  information  used  to  score  the  30 
proposals.  We  see  from  the  table  above  that  the  nature  of  the  proposal  data  is  both 
categorical  and  numerical.  This  proves  challenging  for  conducting  discriminant  analysis 
because  it  is  necessary  that  the  discriminant  score  be  produced  using  numerical  data. 

With  categories  like  Classification  Level,  which  evaluate  a  proposal  based  on  its  status  of 
classified,  secret,  FOUO,  and  none,  there  is  not  a  numerical  representation  of  this 
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measure  as  it  stands.  As  a  result,  each  of  the  evaluation  measures  translates  the  raw  score 
into  their  equivalent  value  by  means  of  the  single  dimensional  value  functions.  The 
additive  value  function  that  describes  this  proposal  evaluation  is  given  below. 

Equation  4  (Dawley,  Marentette,  &  Long,  2008) 

tYAy  .  056vTeneis(xj  “T  - 1  7 6v GapsO^i)  +  -  H  ('iass(X\)  +  - 1 1  2vTimeToCounter(Xj)  +  ■  /  1 V TechPerj(x i)  + 

.056vSuit(xj)  r  .091  Vjnterop(Xi)  •  03 7v-/'(.(:f!^jSjlfx  J  +  ■  0 5 fiv f,-Leiciriime(x j)  +  -087vQpSpurcien(xj)  r 

•  lVworkLoadfai)  +  •05VTrngTime(Xj)  +  .OlSVj 'rngMaturity(Xi) 

The  coefficients  that  precede  the  13  values  are  the  global  weights  for  each  of  the 
respective  values.  The  single  dimensional  value  functions,  denoted  by  v  yaiue_Name(Xi) 
detennine  the  translated  value  for  each  input.  As  such,  vTenets(xj)  is  the  value  earned  for  a 
proposal  that  has  a  particular  number  of  tenets,  x,  .  The  analyst  extracts  the  weighted 
value  contribution  prior  to  conducting  the  discriminant  analysis.  The  image  provided  in 
Figure  3  captures  this  value  contribution  information  for  30  JIEDDO  proposals. 
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Values 


Original 

Global 

0.176 

0.112 

0.110 

0.100 

0.091 

0.087 

0.056 

0.056 

0.056 

0.056 

0.050 

0.037 

0.013 

max 

0.176 

0.112 

0.083 

0.100 

0.091 

0.087 

0.056 

0.042 

0.052 

0.056 

0.050 

0.034 

0.008 

median 

0.049 

0.056 

0.028 

0.070 

0.046 

0.078 

0.042 

0.042 

0.034 

0.028 

0.035 

0.009 

0.008 

min 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.028 

0.020 

0.000 

0.000 

0.000 

0.008 

Figure  3:  Proposal  Score  Breakdown 


The  image  serves  as  a  pictorial  representation  of  how  the  JIEDDO  value  model 
can  be  utilized  to  evaluate  and  justify  proposal  selection  or  rejection.  The  30  proposals 
are  ranked  according  to  score.  The  score  reflects  a  proposal’s  “Potential  to  Defeat  IED” 
based  on  the  13  values  identified  by  JIEDDO  SMEs.  Each  proposal  is  partitioned 
according  to  its  contribution  to  the  overall  score.  The  green-yellow-red  color  scheme  is 
applied  independently  to  each  column  to  show  the  maximum,  median,  and  minimum 
contribution  that  is  made  to  the  overall  score.  Looking  at  Gap  Impact,  for  example, 


30 


proposals  can  achieve  a  maximum  value  contribution  of  0.176  (green),  a  median 
contribution  of  0.049  (yellow),  and  a  minimum  contribution  of  0.000  (red).  The 
contribution  amount  is  reflected  by  the  shading  thereof.  Thus,  Proposals  1,  11,  12,  14, 

15,  18,  19,  all  scored  relatively  well  for  Gap  Impact.  Applying  this  green-yellow-red 
color  scheme  to  each  value  (column)  provides  a  clear  picture  of  how  well  each  proposal 
scored  compared  to  its  competitors  for  each  identified  value.  This  serves  as  a  useful 
pictorial  representation  and  justification  tool  for  organizations  like  JIEDDO,  who  are 
required  to  show  accountability  in  their  decisions. 

After  translating  the  data  into  its  constituent  weighted  value  contribution,  it  is 
possible  to  proceed  with  the  analysis.  Discriminant  Analysis  aids  in  the  prediction  of 
accepting  or  rejecting  proposals  submitted  to  JIEDDO  for  evaluation.  This  research 
applies  Discriminant  Analysis  techniques  to  classify  the  following  two  JIEDDO  proposal 
population  groups:  those  that  are  accepted  and  those  that  are  rejected.  By  creating  a 
discriminant  function  for  each  population  group  utilizing  Equation  3  and  validating  the 
results,  the  author  will  show  that  it  is  possible  to  predict  whether  a  panel  of  decision 
makers  will  likely  accept  or  reject  a  proposal.  In  other  words,  provided  the  knowledge  of 
acceptance  or  rejected  status,  as  well  as  the  variable  infonnation  for  each  of  the  13 
values,  it  is  possible  to  use  the  sample  data  to  create  a  model  to  predict  whether  JIEDDO 
is  likely  to  accept  or  reject  a  given  proposal.  Such  discriminant  functions  serve  as 
indicators  of  JIEDDO’ s  evaluation  thought  process. 

A  confusion  matrix  indicates  the  effectiveness  of  the  discriminant  function  in 
classifying  a  particular  sample  into  the  appropriate  group.  The  confusion  matrix  for  this 
particular  decision  problem  describes  the  proposals  that  were  classified  by  the 
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discriminant  function  in  the  following  ways:  the  proposals  that  should  be  accepted  and 
were  accepted  by  JIEDDO,  the  proposals  that  should  be  accepted  but  were  rejected  by 
JIEDDO,  the  proposals  that  should  be  rejected  and  were  rejected  by  JIEDDO,  and  the 
proposals  that  should  be  rejected  but  were  accepted  by  JIEDDO.  Such  relationships 
among  the  data  are  commonly  depicted  in  the  form  of  the  confusion  matrix  seen  in  Table 
2  below. 

Table  2:  Confusion  Matrix  (Dillon  &  Goldstein,  1984) 


Predicted  Membership 

DF  Categorization 

Accept 

Reject 

Actual 

Accept 

Nic 

Nm 

Membership 

Reject 

N?n 

N2c 

where,  NiC  =  #  of  class  i  classified  correctly 
N,c  =  #  of  class  i  classified  incorrectly 
1  =  Actual  Membership  accept,  2  =  Actual  Membership  reject 

III.C  Lachenbruch’s  Holdout  Procedure 

Lachenbruch’s  Holdout  Procedure  utilizes  the  aforementioned  Discriminant 
Analysis  procedures  to  provide  a  discriminant  function  for  a  particular  observation  or 
holdout.  Although  the  approach  for  developing  the  discriminant  function  and  the 
confusion  matrix  for  Discriminant  Analysis  and  the  Lachenbruch  Holdout  Procedure  are 
the  same,  a  difference  exists  in  splitting  the  original  data  set.  While  Discriminant 
Analysis  uses  a  2/3  training  set  and  1/3  validation  set  to  detennine  the  appropriate 
grouping  for  a  particular  proposal,  Lachenbruch  uses  a  N  -  1  training  set  whereby  N  is 
the  total  number  of  observations  in  the  original  data  set.  The  validation  set  is  comprised 
of  one  holdout.  Thus,  the  approach  is  to  withhold  one  observation  from  the  dataset  and 


32 


run  the  analysis  to  develop  the  discriminant  function  using  the  remaining  observations. 
Upon  identifying  the  appropriate  discriminant  function,  the  analyst  runs  the  single 
holdout  observation  through  each  function  to  detennine  to  which  group  the  holdout 
belongs. 

The  challenges  of  dealing  with  a  small  sample  size  like  that  of  the  30  proposals 
contained  in  the  JIEDDO  model  provide  support  for  using  the  Lachenbruch  Holdout 
Procedure  to  conduct  the  discriminant  analysis.  This  procedure  produces  a  unique 
discriminant  function  for  the  accepted  and  rejected  groups  for  each  of  the  30  proposals. 
As  such,  the  first  ranked  proposal  is  the  first  holdout;  the  procedure  produces  a 
discriminant  function  for  each  acceptance  and  rejection  groups  utilizing  the  data  for  the 
remaining  29  proposals.  Once  the  two  functions  are  created,  proposal  one  is  used  to 
validate  their  effectiveness.  A  discriminant  score  is  produced  using  each  the  accepted 
discriminant  function  and  the  rejected  discriminant  function.  The  proposal  belongs  to  the 
group  whose  discriminant  function  score  is  the  largest.  After  assigning  the  proposal  to  a 
group,  we  verify  the  acceptance  or  rejection  of  the  proposal  to  determine  how  well  the 
function  predicted  the  proposal’s  status.  Next,  the  first  proposal  returns  to  the  dataset,  the 
second  ranked  proposal  exits,  and  the  process  repeats.  The  procedure  creates  a  specific 
set  of  discriminant  functions  for  this  next  ranked  proposal. 

The  iterative  nature  of  Lachenbruch’ s  Holdout  Procedure  allows  for  a  more 
fitting  evaluation  of  a  proposal  in  accordance  with  the  appropriate  accept  or  reject 
determination.  The  confusion  matrix  for  such  a  process  would  contain  one  sample. 
However,  for  the  sake  of  clarity,  the  confusion  matrix  produced  using  this  technique  will 
utilize  the  consolidated  validation  information  for  each  of  the  30  functions. 
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III.D  Sensitivity  Analysis 

The  dynamic  nature  of  the  enemy  to  respond  under  adversity  raises  the  question 
of  how  confident  we  can  be  in  the  JIEDDO  Proposal  Value  Model’s  ability  to 
recommend  advancing  the  proposals  that  will  have  the  greatest  positive  impact.  Success 
relies  heavily  on  selecting  proposals  that  will  adequately  restrict  the  enemies’  ability  to 
react  to  or  counter  our  defense  measures.  By  examining  the  development  of  the  JIEDDO 
Proposal  Value  Model,  specifically  the  weights  placed  on  the  identified  values,  we  will 
provide  confidence  in  its  ability  to  select  and  reject  proposals  appropriately. 

The  current  value  model  resulted  from  SMEs  extracting  JIEDDO ’s  values  from 
within  the  organization.  After  completing  an  extensive  evaluation  process,  the  analysts 
used  affinity  grouping  to  identify  13  values  for  the  decision  problem.  Again,  Table  1 
details  these  values  and  their  respective  definitions  as  originally  defined  by  the  team  of 
analysts. 

The  JIEDDO  team  of  analysts  elicited  weights  for  each  of  the  provided  values 
using  swing  weighting.  Recognizing  the  subjectivity  of  this  process,  it  is  necessary  to 
detennine  the  sensitivity  of  the  overall  rank  order  produced  via  weight  changes  within  the 
additive  value  function.  By  demonstrating  the  soundness  of  the  suggested  model  results, 
JIEDDO  decision  makers  will  be  able  to  utilize  the  model  confidently  as  a  filter  for 
selecting  or  rejecting  the  proposals.  Furthennore,  the  JIEDDO  model  will  serve  as  a 
useful  tool  in  providing  detailed  feedback  to  the  applicant  on  where  their  proposal 
succeeded  (or  failed)  to  meet  pre-designated  JIEDDO  values. 
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In  order  to  demonstrate  the  robustness  of  the  model,  it  is  necessary  to  investigate 
the  sensitivity  of  the  weights  themselves  on  multiple  levels.  We  know  from  traditional 
sensitivity  analysis  that  it  is  common  to  investigate  one-way  and  two-way  sensitivity 
analyses.  This  involves  observing  the  rank  order  as  one  or  two  values  are  adjusted 
keeping  the  remaining  values  proportionately  constant.  Such  analysis  has  the  potential  to 
provide  insight  into  the  more  dominant  values  in  a  given  decision  problem  (Bauer, 
Parnell,  &  Meyers,  1999).  However,  it  is  equally  important  to  expand  sensitivity  analysis 
to  include  those  that  consider  adjusting  three  or  more  weights  at  a  time,  thus  conducting 
n-way  sensitivity  analysis  (Marks,  2008). 

The  previous  research  conducted  by  Hunter  Marks  compared  a  series  of  five 
different  n-way  sensitivity  analysis  techniques  and  investigated  their  effectiveness. 

Marks  utilized  mathematical  programming  techniques  to  detennine  the  minimum  weight 
change  that  can  be  made  across  all  weights  while  preserving  the  rank  of  a  given 
alternative.  Thus,  he  examined  the  distance  of  a  given  alternative  from  the  remaining 
population  of  alternatives  in  n-space  by  using  the  following  five  math  programming 
techniques:  least  squares,  1-norm,  2-norm,  oo-norm,  and  percent  change  metrics. 

Although  he  recommended  utilizing  the  1-norm  for  his  particular  Course  of  Action  Value 
Model,  he  recognized  that  each  of  the  five  metrics  is  useful.  As  such,  his  research 
suggests  that  it  is  permissible  to  choose  any  one  metric  to  develop  and  conduct  sensitivity 
analysis  for  the  JIEDDO  value  model. 

This  research  employs  the  2-norm  to  detennine  the  minimum  distance  in  the 
weight  space  between  a  pair  of  proposals  such  that  their  overall  value  scores  are  equal. 
Thus,  we  are  looking  to  find  a  new  set  of  weights  such  that  these  new  weights,  when 
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applied  to  the  additive  value  function  (replacing  the  original  global  weights),  produce 
equal  scores  for  the  two  proposals  being  compared.  For  example,  consider  proposal 
number  one  and  proposal  number  four.  The  objective  is  to  detennine  the  minimum 
weight  change  such  that  the  value  of  proposal  one  equals  that  for  proposal  four.  The 
original  global  weights  are  perturbed  simultaneously  to  produce  a  new  set  of  weights  that 
will  achieve  this  result.  We  have  the  following  problem  fonnulation. 

k 

Problem  3.1  1  (Marks,  2008)  Minimize  ^(ff.-w.)2 

i= 1 

subject  to: 

A- 

yj  (vA  (x. )  -vs  (x.  ))vv  =0  VT  ^  B 

i=l 

ZlF=  1 

2>,.= i 

j=i 

0  <  w.,  W.  <  1  Vi  =  l...k 

where: 

W.  =  the  initial  weights  defined  by  the  decision  maker 
w.  =  the  weights  found  that  minimize  the  measure 
vA  (jc  )  =  value  score  of  attribute  i  for  alternative  A 

Problem  3.11  illustrates  measuring  the  sensitivity  of  proposals  utilizing  a 
pairwise  comparison  method.  Thus,  we  seek  a  new  set  of  weights  that  will  force  the 
value  of  a  given  proposal  A  to  be  equal  to  that  of  B.  The  problem  fonnulation  aims  to 
minimize  the  weight  change  from  the  original  global  weights  to  the  new  weights  for  each 
of  the  values.  Additionally,  each  set  of  weights  must  sum  to  one  and  exhibit 
non-negativity.  Thus,  this  research  examines  the  distance  between  two  different 
proposals  in  n-space,  where  n  is  the  number  of  identified  values  for  the  decision  problem. 
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While  comparing  the  two  proposals,  we  recognize  that  the  new  set  of  weights  will 
ensure  that  they  have  the  same  value,  the  new  set  of  weights  will,  correspondingly,  affect 
the  value  of  the  remaining  twenty  eight  proposals.  As  a  result,  we  acknowledge  rank 
preservation  of  the  remaining  proposals  is  not  guaranteed  when  solving  the  optimization 
problem  for  a  specific  pair  of  proposals. 

This  research  applies  the  2-norm  math  programming  technique  to  each  proposal. 
As  such,  for  a  set  of  m  different  proposals,  this  optimization  problem  will  be  applied  a 
total  of  m  times.  However,  since  this  research  makes  pairwise  comparisons,  it  is  given 
that  the  minimum  distance  from  proposal  A  to  proposal  B  is  the  same  as  saying  the 
minimum  distance  from  proposal  B  to  proposal  A.  In  addition,  the  minimum  distance 
from  a  given  proposal  to  itself  is  zero.  The  number  of  unique  pairwise  comparisons  can 
be  reduces  to  (m  -m)/2  or  m(m-l)/2. 

In  order  to  consolidate  the  distance  information  identified  from  employing  the 
2-norm  adequately,  it  is  possible  to  construct  an  m  x  m  distance  matrix  where  m  is  the 
number  of  proposals  whose  rank  change  the  author  wishes  to  observe.  This  research  uses 
the  thirty  proposals  from  the  currently  available  data,  thereby  resulting  in  435  unique 
pairwise  comparisons.  The  matrix  contains  the  distance  between  two  different  proposals 
for  all  the  pairwise  comparisons  of  the  proposals’  values.  The  distance  matrix  produced 
using  the  Problem  3.11  formulation  will  resemble  that  seen  in  Figure  4. 
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Figure  4:  Distance  Matrix 


Examining  the  distances  between  proposals  in  n-space  where  n  is  the  number  of 
variables  under  consideration  (13  for  the  case  of  JIEDDO)  will  ultimately  reveal 
information  about  the  sensitivity  of  the  weights  themselves.  If  a  weight  change  is  made, 
we  recognize  that  it  is  as  minimal  as  possible  as  to  meet  the  desired  objective.  Given  that 
a  weight  change  does  occur,  we  know  w;  reflects  the  newly  generated  weights.  From  Wj, 
we  utilize  the  weighted  value  function  previously  defined  in  Equation  4  with  the  new 
weights  in  order  to  verify  that  such  a  calculation  will  in  fact  produce  equal  scores  for 
proposals. 

The  distance  matrix  formulation  allows  us  to  observe  how  far  apart  proposals  are 
from  each  other,  thereby  characterizing  the  distances  between  distinct  proposals. 
Conversely,  a  similarity  matrix  depicts  the  similarities  between  a  given  pair  of  proposals 
(Bauer  K.  W.,  2008).  The  similarity  matrix  is  constructed  using  the  formulation  as 
shown  in  Figure  5. 
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Let  S  be  the  similarity  matrix  produced  using  the  distance  matrix  D. 


Equation  5  (Bauer  K.  W.,  2008) 


The  similarity  matrix  becomes: 


l/(l+0) 

1/(1+D,,2) 

i/O+Oi,) 

1/(1+D,,3o) 

l/Q+D2.l) 

l/(l+0) 

i/O+D^) 

1/(1+D2.3o) 

l/(l+0) 

l/(l+0) 

i/d+EW 

l/(l+0) 

Figure  5:  Similarity  Matrix 


The  similarity  matrix  in  Figure  5  resembles  that  of  a  correlation  matrix.  In  fact, 
the  correlation  matrix  has  often  been  characterized  as  a  similarity  matrix  because  it 
applies  the  pairwise  comparison  technique  to  depict  how  closely  related  a  given  set  of 
variables  are  to  each  other  (Bauer  K.  W.,  2008).  As  such,  perfectly  correlated  variables 
have  a  correlation  factor  equal  to  one  while  those  variables  that  are  independent  of  each 
other  possess  a  correlation  factor  equal  to  zero  (Bauer  K.  W.,  2008).  Once  the  similarity 
matrix  is  fonned  from  the  distance  matrix,  this  research  will  investigate  the  sensitivity  of 
the  proposals  by  employing  Factor  Analysis  to  extract  more  information  about  the 
population  set  of  interest.  More  specifically,  the  author  will  use  Factor  Analysis  to 
examine  clustering  among  the  various  JIEDDO  proposals. 

According  to  Dillon  and  Goldstein,  Factor  Analysis  seeks  to  “simplify  complex 
and  diverse  relationships  that  exist  among  a  set  of  observed  variables  by  uncovering 
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common  dimensions  or  factors  that  link  together  the  seemingly  unrelated  variables” 
(Dillon  &  Goldstein,  1984).  Factor  Analysis  thereby  relates  to  this  research’s  purpose  to 
detennine  the  common  relationships  among  seemingly  different  proposals  for  the 
JIEDDO  value  model  proposal  evaluation  process.  Dillon  and  Goldstein  provide  the 
following  formulation  for  the  factor-analytic  model. 

Equation  6  (Dillon  &  Goldstein,  1984) 


X  =  A f  +  e 


where: 

X  =  p-dimensional  vector  of  observed  responses,  X’  =  (xi,  X2,  X3,  X4, . ,  xp) 

/  =  q-dimensional  vector  of  unobservable  variables  called  common  factors, 

/  =  (/!, /2,/3,../q) 

e  =  p-dimensional  vector  of  unobservable  variables  called  unique  factors, 
e  (c  1 ,  e2,  e3,...  cp) 

A  =  pxq  matrix  of  unknown  constants  called  factor  loadings 


The  number  of  retained  factors  and  their  underlying  variable  contribution  is 
detennined  using  the  loadings  matrix.  Dillon  and  Goldstein  describe  the  loadings 
accordingly,  stating  “they  tell  us  which  variables  are  involved  in  what  factor  and  to  what 
degree”  (Dillon  &  Goldstein,  1984).  The  loading  matrix  for  the  similarity  matrix  is 
calculated  using  Equation  7. 

Equation  7  (Bauer  K.  W.,  2008) 

L  —  \levalue  *  e vector 

where: 

L  =  loading  matrix 
£  value  eigenvalue 

evector  =  eigenvector 
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From  the  loading  matrix,  this  research  will  use  a  heuristic  to  determine  the 
“pattern  matrix”  (Dillon  &  Goldstein,  1984).  This  procedure  necessitates  starting  with 
the  first  variable  and  the  first  factor  and  moving  horizontally  across  the  factors  of  the 
loading  matrix  and  selecting  the  loading  with  the  largest  positive  or  negative  contribution 
to  the  given  factor  (Dillon  &  Goldstein,  1984).  Next,  we  consider  the  second  row  (or 
proposal  for  this  research)  looking  for  the  greatest  positive  or  negative  contribution  to  a 
given  factor  and  circling  it  appropriately.  This  process  continues  for  each  of  the  30 
proposals  that  comprise  the  similarity  matrix  used  in  this  analysis. 

Dillion  and  Goldstein  describe  the  variable  loading  examination  process,  “After 
all  the  variables  have  been  considered,  examine  each  circled  loading  for  significance” 
(1984).  They  suggest  assessing  the  statistical  or  practical  significance  as  it  applies.  As 
such,  the  statistical  significance  as  it  applies  to  small  sample  sizes  like  that  for  JIEDDO 
would  have  to  be  greater  than  +/-  0.30  in  order  to  be  deemed  significant  (Dillon  & 
Goldstein,  1984).  A  practical  evaluation  of  significance  would  imply  setting  a  rule  for  a 
minimum  amount  of  accountable  variance  for  a  given  factor.  For  this  research,  we  will 
assess  the  significance  using  the  statistical  significance  evaluation  suggested.  Once  this 
process  is  complete,  the  examination  of  remaining  “insignificant”  variables  will  be 
reviewed  and  assessed  to  detennine  their  relevance  for  a  given  factor.  Lastly,  the  author 
will  extrapolate  meaning  from  the  pattern  of  factor  loadings  by  assigning  a  name  to  the 
factor  that  incorporates  the  variables  reflected  thereof. 

The  loading  matrix  produced  using  this  method  is  a  “particular  interpretation”  of 
the  data  (Dillon  &  Goldstein,  1984).  Once  this  process  is  complete  and  the  loadings  have 
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been  calculated  in  their  original  fonn,  it  is  then  possible  to  rotate  the  loading  matrix. 
Rotating  the  factor  loadings  matrix  permits  us  to  view  the  factors  from  a  different 
perspective.  This  perspective  is  allows  us  to  interpret  the  factors  from  a  varying  degree 
of  directions. 

The  three  orthogonal  rotation  methods  available  to  use  for  this  analysis  consist  of 
varimax,  quartimax,  and  equimax  methods  (Dillon  &  Goldstein,  1984).  The  objective  of 
the  varimax  method  is  to  rotate  the  factors  in  a  manner  that  will  achieve  the  largest 
squared  factor  loadings  for  a  given  factor.  The  quartimax  method  rotates  the  factors  by 
spreading  the  variables  so  that  there  is  a  one  for  one  ratio  between  a  given  variable  and  a 
factor.  Dillon  and  Goldstein  describe  this  method  as  that  which  is  very  difficult  to 
accomplish  (1984).  The  third  method,  the  equimax  method,  consists  of  rotating  until  a 
“simple  structure  with  respect  to  the  rows  and  columns”  is  achieved  (Dillon  &  Goldstein, 
1984).  The  varimax  method  is  the  most  common  of  the  rotation  techniques  and  this 
research  uses  it  to  determine  key  factors  for  the  JIEDDO  model. 

The  clusters  produced  via  Factor  Analysis  permit  us  to  make  additional 
observations  about  the  proposal  groupings  themselves.  For  example,  suppose  that  the 
Factor  Analysis  generates  three  clusters  of  proposals  and  each  proposal  is  then  grouped 
according  to  the  cluster  to  which  it  belongs.  Thus,  we  have  a  set  of  proposals  in  the  first 
cluster,  a  (different)  set  of  proposals  in  the  second  cluster,  and  the  remaining  proposals 
clustered  in  the  third.  After  clearly  identifying  the  clusters,  it  is  important  to  look  for 
common  characteristics  among  the  proposals  that  fall  within  a  given  cluster.  There  is  a 
reason  why  such  proposals  are  grouped;  the  research  team  must  identify  these  reasons.  A 
cluster  of  proposals  that  rank  in  the  top  third  with  respect  to  their  overall  value  score 
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indicates  similarities  among  the  proposals  themselves.  Correspondingly,  accepting  all 
proposals  except  one  within  a  cluster  suggests  that  there  is  inconsistency  within  the 
evaluation  process. 

Using  this  information,  the  research  will  utilize  the  original  distance  matrix  to 
determine  the  average  distance  between  proposals  within  a  given  cluster  as  well  as  find 
the  average  distance  between  clusters.  By  determining  the  average  distance  within  and 
between  clusters,  this  research  seeks  to  demonstrate  the  significance  of  being  in  one 
cluster  as  opposed  to  another.  Figure  6  shows  a  pictorial  view  of  this  example. 


Figure  6:  Sample  Cluster  Analysis 


This  sample  clustering  analysis  illustrates  the  relative  distances  within  the  weight 
space  both  within  and  between  clusters.  As  such,  we  see  that  the  average  distance 
between  proposals  within  a  given  cluster  is  much  smaller  than  that  seen  between  the 
clusters  themselves.  Take  cluster  two,  for  example.  We  see  that  the  average  distance 
between  any  two  proposals  within  in  the  set  (proposal  10  to  proposal  22)  is  0.03. 


43 


However,  the  average  distance  between  clusters  one  and  two,  and  two  and  three  is  0.120 
and  0.140,  respectively. 

Alongside  the  overview  of  proposals  sensitivity  provided  by  observing  clustering, 
additional  insight  is  gained  by  further  investigating  the  new  weights  generated  from  the 
minimum  weight  change  analysis.  This  research  will  use  imaging  to  illustrate  the 
analysis  problem  by  depicting  the  weight  changes  that  produce  the  minimum  distance 
such  that  two  given  proposals  considered  indifferent  in  value. 

Image  comparisons  are  provided  for  both  the  percent  weight  change  with  respect 
to  the  original  global  weights  and  the  actual  weight  change  values  themselves.  An  image 
is  generated  for  each  proposal.  The  images  are  formed  by  utilizing  the  new  weights 
provided  via  the  pairwise  comparison  metric  described  in  Problem  3.1.1  and  is 
exemplified  by  the  15th  ranked  proposal  in  the  set.  A  30  by  13  matrix  is  produced 
whereby  each  row  contains  the  new  weights  generated  for  each  of  the  13  values  when  the 
15th  ranked  proposal  is  compared  to  a  proposal  whose  rank  designation  is  equal  to  that  of 
the  row  in  which  it  resides.  From  this  information,  the  percent  weight  change  is 
calculated  with  respect  to  the  original  global  weights  for  each  value  and  thereby 
providing  the  positive  or  negative  change  for  each  of  the  13  weights. 

Once  this  is  complete,  the  imaging  is  produced  via  a  coloring  effect  which  is 
coded  using  the  standard  red  and  green  metric.  The  largest  positive  and  negative  weight 
changes  are  identified  by  using  a  percentile  metric  where  all  of  the  weight  changes  are 
compared  with  the  most  negative  weight  change  falling  at  the  zero  percentile  and  the 
most  positive  weight  change  occurring  at  the  100th  percentile.  As  such,  we  generally  find 
that  dark  green  indicates  a  positive  percent  weight  change  and  red  indicates  a  negative 
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weight  change.  A  different  color  shade  is  attributed  to  every  fifth  percentile  captured. 
This  imaging  technique  is  applied  to  both  the  percent  weight  change  values  as  well  as  the 
actual  weight  change  for  all  of  the  proposals  under  consideration.  An  example  of  the 
imaging  technique  utilized  in  this  research  is  shown  in  Figure  7. 


Proposal 
Ranked  # 


2 

3* 

4 

5* 

6 


8* * 

9* 

10 

11 
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13* 

14 
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18* 
19* 
20* 
21* 
22* 
23* 
24* 
25* 
26* 
27* 
28 
29* 
30* 


Original 

Global 

Weights 


Average  % 
Weight 
Change 


Figure  7:  Proposal  #15  Percent  Change 


Figure  7  above  depicts  the  image  generated  for  proposal  15.  The  columns  of  the 


image  represent  the  13  values  arranged  according  from  largest  to  smallest  global  weight. 


Each  value’s  respective  global  weight  is  shown  directly  below  the  generated  image.  The 


percent  weight  change  from  the  original  global  weights  to  the  new  set  of  weights  that 


produces  equal  value  to  that  of  proposal  15  is  shown  for  each  row  of  the  matrix.  For 


example,  consider  the  comparison  of  proposal  15  to  proposal  1.  Based  on  the 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-2  -7  10  -12  -19  -14  25  19  8  -4  1  1  106 


*  Indicates  JIEDDO  rejected  proposal 
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characteristics  that  define  proposal  15,  the  weights  for  Gap  Impact,  Work  Load, 
Interoperability,  Fielding  Timeline,  Suitability,  and  Technical  Risk  would  need  to 
decrease  simultaneously  for  proposal  1 5  to  yield  a  total  score  equal  to  that  of  proposal  1 
as  shown.  Additionally,  the  weights  for  Technical  Performance,  Program  Maturity, 
Tenets  Impacted,  Classification,  Operations  Burden,  Training  Time,  as  well  as  Time  to 
Counter  would  need  to  increase.  From  the  image  shading,  we  observe  that  Technical 
Performance,  Program  Maturity,  and  Technical  Risk  are  among  those  with  the  largest 
percent  weight  change  (as  denoted  by  the  color  intensity).  Furthermore,  we  see  from  the 
suggested  percent  weight  changes  that  proposal  15  exhibits  value  strength  when 
compared  to  the  top  ranked  proposal  in  Program  Maturity,  Technical  Perfonnance, 

Tenets  Impacted,  and  Classification  level.  Conversely,  weaknesses  surface  in  Technical 
Risk,  Gap  Impact,  and  Interoperability. 

Looking  at  proposal  15’s  neighbors,  proposal  14  and  proposal  16,  we  notice  only 
slight  variations  among  varying  values.  These  small  percent  weight  changes  indicate 
weight  sensitivity  when  looking  at  proposals  most  near  that  of  the  observed  proposal. 

We  gain  additional  insight  by  generating  this  image  under  a  percent  weight 
transfonnation.  The  transformed  image  is  produced  by  taking  the  average  percent  weight 
change  for  each  of  the  respective  values  and  sorting  them  from  most  positive  to  most 
negative  average  percent  weight  change.  This  is  shown  for  proposal  15  in  Figure  8. 
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Proposal 
Ranked  # 


Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.013  0.056  0.056  0.110  0.056  0.037  0.050  0.176  0.056  0.112  0.100  0.087  0.091 


106  25  19  10  8  1  1  -2  -4  -7  -12  -14  -19 


*  Indicates  JIEDDO  rejected  proposal 


Figure  8:  Proposal  #15  Percent  Change  (Sorted  Vertically) 

Based  on  the  average  percent  weight  change  metric,  the  figure  illustrates  that  the 
most  positive  weight  change  occurs  for  Program  Maturity  and  Classification.  In  general, 
the  most  negative  weight  change  occurs  for  Work  Load,  Operations  Burden,  and 
Interoperability.  Observing  the  percent  weight  changes  in  this  manner  allows  for  a 
broader  view  of  which  values  dominate  a  given  proposal’s  overall  score. 

The  proposal  evaluation  imaging  process  can  be  taken  a  step  further  by  analyzing 
the  images  for  the  actual  weight  changes.  Actual  weight  change  imaging  allows  us  to  see 
by  how  small  or  large  a  weight  fluctuation  is  occurring.  Figure  9  and  Figure  10  depict 
the  actual  weight  changes  that  are  occurring  for  proposal  15  for  each  pairwise 
comparison. 
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Original  Global 
Weights 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


Average  weight 
change 


-0.003  -0.008  0.011  -0.012  -0.017  -0.012  0.010  0.014  -0.002  0.005  0.000  0.000  0.014 


*  Indicates  JIEDDO  rejected  proposal 


Figure  9:  Proposal  #15  Weight  Change 


Proposal  Ranked 
# 

1 

2 

3* * 

4 

5* 

6 
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8* 

9* 
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14 
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16 
17 
18* 

19* 

20* 

21* 

22* 

23* 

24* 

25* 

26* 

27* 

28 

29* 

30* 


Original  Global 
Weights 


Average  weight 
change 


0.056  0.013  0.110  0.056  0.056  0.050  0.037  0.056  0.176  0.112  0.100  0.087  0.091 


0.014  0.014  0.011  0.010  0.005  0.000  0.000  -0.002  -0.003  -0.008  -0.012  -0.012  -0.017 


’  Indicates  JIEDDO  rejected  proposal 


Figure  10:  Proposal  #15  Weight  Change  (Sorted  Vertically) 
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The  percent  weight  change  and  actual  weight  change  images  for  Proposal  15 
exhibit  similar  patterns  in  behavior  as  shown  in  the  side  by  side  view  below.  The  weight 
on  Gap  Impact  would  need  to  decrease  while  the  weights  for  Time  to  Counter  and 
Technical  Performance  increase  in  order  for  a  rank  change  to  occur  between  proposal  15 
and  those  ranked  above  it.  Conversely,  the  weight  for  Gap  Impact  would  increase  while 
the  weights  for  Time  to  Counter  and  Technical  Performance  decrease  for  proposal  15  to 
move  down  in  the  ranks.  Thus,  we  generally  observe  a  horizontal  mirroring  effect  for  the 
percent  weight  change  and  actual  weight  change  images  for  a  given  proposal. 


Figure  11:  Proposal  15  Percent  Change 


Figure  12:  Proposal  15  Weight  Change 


Again,  comparing  the  transformed  (sorted)  images  for  each  percent  and  actual 
change,  we  observe  a  vertical  mirroring  affect. 
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Figure  13:  Proposal  15  Percent  Change 
(Sorted  Vertically) 


Figure  14:  Proposal  15  Weight  Change 
(Sorted  Vertically) 


The  transformed  images  shed  light  on  the  average  positive  or  negative  weight  change. 

For  proposal  15,  the  average  change  among  nearly  all  proposals  is  positive  for 
Classification  and  Program  Maturity.  The  majority  of  proposal  comparisons  for  proposal 
1 5  indicate  decrease  weight  changes  for  Interoperability. 

III.E  Summary 

This  chapter  reviewed  the  methodology  employed  in  validating  the  model’s 
ability  to  predict  JIEDDO’s  decision  making  though  process  well  as  the  decision 
problem’s  sensitivity  to  simultaneous  perturbations  in  the  model’s  weights.  More 
specifically,  the  use  of  multivariate  techniques  such  as  Lachenbruch  Holdout  Procedure 
for  Discriminant  Analysis  was  discussed  in  detail  to  develop  a  function  that  would  reflect 
the  acceptance  and  rejection  proposal  populations.  Factor  Analysis  was  utilized  in 
combination  with  math  programming  techniques  to  collect  additional  insight  regarding 
the  sensitivity  of  the  weights  for  the  decision  problem.  The  results  for  this  applied 
methodology  as  it  pertains  to  the  JIEDDO  value  model  are  contained  in  the  following 
chapter. 
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IV.  Results  and  Analysis 


IV.A  Introduction 

This  chapter  explains  the  results  of  the  research  involving  the  validation  and 
weight  sensitivity  for  the  JIEDDO  value  model.  In  particular,  it  provides  detailed 
discussion  and  analysis  of  the  model’s  robustness  based  on  the  methodology  described  in 
Chapter  3.  First,  the  author  will  investigate  the  model’s  ability  to  predict  the  JIEDDO ’s 
decision  process  by  applying  the  Lachenbruch  Holdout  Procedure.  Second,  the  author 
will  evaluate  the  sensitivity  of  the  model  under  simultaneous  weight  changes  to  indicate 
the  level  of  confidence  in  the  model’s  results. 

IV.B  Lachenbruch’s  Holdout  Procedure 

The  iterative  nature  of  the  Lachenbruch  Holdout  Procedure  facilitates  splitting  the 
original  data  set  in  a  manner  that  accounts  for  a  larger  number  of  observations.  A 
discriminant  function  is  determined  for  the  accept  group  as  well  as  the  reject  group  using 
a  training  sample  comprising  of  N-l  observations  (where  N  is  the  total  number  of 
observations).  The  remaining  observation,  or  holdout,  is  then  validated  using  the 
discriminant  functions  produced  by  the  training  data.  For  the  30  proposals  used  in  this 
research,  the  training  group  comprises  of  29  samples  and  the  validation  group  comprises 
of  1  sample.  A  separate  discriminant  function  is  produced  for  each  proposal  utilizing  the 
remaining  29. 

A  confusion  matrix  can  be  provided  for  each  one  of  the  validation  observations, 
or  holdouts.  However,  for  the  sake  of  clarity,  the  consolidated  result  produced  after 
applying  this  technique  to  each  of  the  30  JIEDDO  proposal  evaluation  processes  is  shown 
in  the  confusion  matrix  in  the  Table  3. 
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Table  3 


Actual 

JIEDDO 

Decision 


Accept 

Reject 


Predicted  Membership 
DF  Categorization 


Accept 


Reject 


13 

0 

0 

17 

The  associated  apparent  error  rate  (APER)  is: 


A  PER  = 


*1C  +  ^2C 


nx  +n2 


where,  NiC  =  #  of  class  i  classified  correctly 
N,c  =  #  of  class  i  classified  incorrectly 
1  =  Actual  Membership  accept,  2  =  Actual  Membership  reject 
n i  =  total  number  of  samples  in  group  1 
n2  =  total  number  of  samples  in  group  2 


The  ability  of  a  discriminant  function  to  predict  the  acceptance  or  rejection  of  a 
proposal  that  has  been  withheld  from  the  population  is  promising.  The  100%  hit  rate 
justifies  using  the  Lachenbruch  Holdout  Procedure  to  predict  the  status  of  a  proposal. 
Furthennore,  it  serves  as  motivation  that  the  discriminant  functions  and  JIEDDO  decision 
are  compatible.  Thus,  this  research  proves  it  is  possible  to  create  a  discriminant  function 
that  predicts  JIEDDO’ s  decision  process.  However,  the  question  remains  as  to  whether 
or  not  their  decision  process  is  consistent  with  their  values  as  expressed  in  the  value 
model. 

This  inconsistency  is  demonstrated  by  the  proposal  scores  and  ranks  in  Table  4. 
JIEDDO  rejected  five  proposals  in  the  top  half.  Meanwhile,  they  accepted  proposal  V, 
which  ranked  third  from  the  bottom.  This  begs  the  question  as  to  the  model’s  validation. 
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More  specifically,  it  raises  the  question  of  how  confident  should  we  be  in  the  model’s 
ability  to  reflect  the  decision  based  on  the  predefined  specified  values. 


Table  4:  Proposal  Value  Model  Results 


*  Indicates  JIEDDO  rejected  proposal 


Further  investigation  proves  that  all  four  rejected  proposals  in  the  top  half  are 
research  and  development  (R&D)  proposals.  As  a  result,  F*,  E*,  J*,  and  B*  scored 
“artificially  high”  in  the  areas  of  Time  to  Counter,  Technical  Performance,  Suitability, 
Interoperability,  Operations  Burden,  Work  Load,  and  Training  Time  (Dawley, 
Marentette,  &  Long,  2008). 

A  team  of  AFIT  scored  all  of  the  proposals  provided  for  this  research.  In  regard 
to  the  R&D  proposals,  according  to  the  analysts  that  scored  the  proposals,  “their 
deliverable  was  a  research  paper  instead  of  a  prototype  system.”  (Dawley,  Marentette,  & 
Long,  2008)  This  fact  made  scoring  very  difficult  for  the  aforementioned  evaluation 
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measures.  For  Interoperability,  Dawley,  Marentette,  and  Long  define  this  measure  as, 
“The  degree  to  which  a  system  fits  into  the  existing  network  architecture,  whether  it  can 
exchange  data  with  supporting  and  supported  systems,  and/or  whether  it  can  perfonn  its 
task  without  negatively  impacting  friendly  assets”  (2008).  Based  on  this  definition,  it 
would  be  easy  to  say  that  the  R&D  paper  topics  would  not  “negatively  impact  friendly 
assets”  because  it  is  not  an  actual  system.  As  a  result,  the  analysts  gave  it  the  most 
valuable  score  for  this  particular  score.  In  fact,  all  of  the  proposals  received  the  most 
valuable  score  for  each  of  the  aforementioned  values  as  they  apply  to  the  situation.  It  is 
for  this  reason,  that  the  R&D  proposals  produced  inflated  value  model  scores.  The  nature 
of  the  R&D  proposals  end  product  being  that  of  a  research  paper,  vice  a  C-IED 
“prototype  system”,  makes  utilizing  the  model  for  their  evaluation  inappropriate 
(Dawley,  Marentette,  &  Long,  2008).  As  a  result,  this  research  cannot  support  the 
evaluation  of  R&D  proposals  using  the  current  JIEDDO  Proposal  Value  Model.  A  new 
model  is  necessary  for  the  evaluation  of  R&D  proposals.  The  aforementioned  four 
rejected  proposals  in  addition  to  one  that  had  been  accepted  (DD)  were  recommended  for 
removal  from  model  consideration  (Dawley,  Marentette,  &  Long,  2008). 
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Table  5:  Proposal  Value  Model  Results  (R&D  removed) 


From  Table  5  above  we  see  two  outliers,  P*  and  V.  P*  was  rejected  when  four 


proposals  scoring  below  it  were  accepted.  V  was  accepted  although  it  ranked  third  from 


the  bottom.  By  taking  a  closer  look  at  these  two  proposals,  we  look  to  gain  insight  into 


the  nature  of  such  occurrences. 
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Table  6:  Anomaly  Investigation 


Value  Name 
Tenets  Impacted 
Gap  Impact 
Classification 
Time  to  Counter 
Technical  Performance 
Suitability 
Interoperability 
Technical  Risk 
Fielding  Timeline 
Operations  Burden 
Work  Load 
Training  Tme 
Program  Maturity 
Total  Score 


Weighted  Contribution  to  Value  Function 
P'  (rejected)  V  (accepted) 


Ke 

y 

Value  contribution 

Color  Identifier 

Top  third 

Middle  third 

Bottom  third 

|  rejected  proposal  | 

Both  proposal  P*  and  proposal  V  are  presented  in  Table  6:  Anomaly 
Investigation.  All  13  values  are  listed  to  provide  information  as  to  how  well  each 
proposal  scored  for  each  category.  The  value  contribution  to  the  total  score  is  coded 
using  a  green-yellow-red  metric.  Proposals  that  possess  highly  valuable  characteristics 
are  shaded  green,  mildly  valuable  are  yellow,  and  minimally  valuable  in  red.  For 
example,  Gap  Impact  contains  9  different  measures:  Gl,  G2,  G3,  G4,  G5,  G6,  G7,  G8, 
and  none,  where  Gl  is  considered  the  most  desired  category  and  is  ranked  first.  If  a 
proposal  meets  a  Gap  Impact  at  the  G1-G3  level,  it  receives  the  green  shading. 
Conversely,  if  it  falls  between  G7  -  none,  it  receives  the  red  shading.  This  metric  thus 
reveals  that  both  proposal  P*  and  V  scored  poorly  for  Gap  Impact. 

When  compared  to  proposal  V  (accepted),  proposal  P*  actually  scores  better  in 
the  following  areas:  Tenets  Impacted,  Time  to  Counter,  Technical  Risk,  Fielding 
Timeline,  Operations  Burden,  and  Work  Load.  However,  proposal  P*  scores  poorly  for 
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Training  Time  (0.000)  as  compared  to  proposal  V  (0.035).  This  observation  may  be  the 
reason  for  JIEDDO’s  decision  to  reject  the  proposal.  However,  it  still  leaves  us 
wondering  why  such  a  low  scoring  proposal  like  proposal  V  would  be  accepted  as  it  does 
not  meet  any  of  the  desired  Gap  Impact  requirements. 

From  these  observations,  we  are  able  to  make  the  following  conclusions.  First, 
the  value  model  appears  valid  for  non-R&D  proposals.  However,  it  is  necessary  to  check 
the  sensitivity  of  the  proposals  to  simultaneous  weight  changes.  Conducting  sensitivity 
analysis  may  explain  why  Proposal  P*  was  rejected.  Second,  implementing  a  value 
model  to  aid  the  decision  making  process  is  necessary  for  making  consistent  and 
justifiable  decisions.  The  decision  to  accept  proposal  V  is  inconsistent  with  previous 
JIEDDO  decisions. 

IV.C  Sensitivity  Analysis 

Sensitivity  analysis  is  conducted  based  on  the  ranking  assessment  technique 
outlined  in  chapter  three.  The  minimum  distance  matrix  was  produced  based  on  the  L2 
nonn  calculation.  Additionally,  the  distance  matrix  produced  via  the  optimization 
technique  was  utilized  to  construct  the  similarity  matrix  for  this  problem.  Both  the 
distance  and  similarity  matrix  for  the  thirty  JIEDDO  proposals  under  evaluation  are 
provided  in  Appendix  A. 

The  author  conducts  Factor  Analysis  on  the  similarity  matrix  in  order  to 
investigate  possible  clustering  among  proposals.  Clustering  among  proposals  indicates 
observed  commonalities  among  proposals.  Utilizing  the  heuristic  outlined  by  Dillon  and 
Goldstein,  the  author  identified  three  clusters. 
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Table  7:  3  Factor  Analysis  (Rotated) 


Proposal  Ranked 

Factor  1 

Factor  2 

Factor  3 

Score 

1 

0.358 

-0.461 

0.711 

0.822 

2 

0.439 

-0.300 

0.849 

0.727 

3* 

0.500 

-0.435 

0.716 

0.683 

4 

0.522 

-0.377 

0.746 

0.672 

5* 

0.521 

-0.443 

0.701 

0.672 

6 

0.598 

-0.459 

0.636 

0.620 

7 

0.718 

-0.350 

0.587 

0.613 

8* 

0.669 

-0.496 

0.523 

0.599 

9 

0.743 

-0.414 

0.517 

0.584 

10 

0.681 

-0.521 

0.491 

0.576 

11 

0.668 

-0.515 

0.525 

0.565 

12 

0.693 

-0.504 

0.492 

0.563 

13* 

0.770 

-0.459 

0.446 

0.561 

14 

0.692 

-0.531 

0.473 

0.561 

15 

0.746 

-0.479 

0.439 

0.555 

16 

0.686 

-0.524 

0.484 

0.554 

17 

0.732 

-0.491 

0.459 

0.539 

18* 

0.687 

-0.541 

0.461 

0.539 

19* 

0.670 

-0.557 

0.463 

0.528 

20* 

0.630 

-0.629 

0.422 

0.502 

21* 

0.597 

-0.636 

0.460 

0.491 

22* 

0.623 

-0.619 

0.443 

0.488 

23* 

0.584 

-0.653 

0.435 

0.477 

24* 

0.518 

-0.703 

0.441 

0.447 

25* 

0.488 

-0.740 

0.457 

0.420 

26* 

0.484 

-0.807 

0.317 

0.401 

27* 

0.452 

-0.827 

0.305 

0.387 

28 

0.447 

-0.804 

0.329 

0.367 

29* 

0.435 

-0.868 

0.252 

0.364 

30* 

0.159 

-0.800 

0.476 

0.170 

Varimax 

10.772 

10.235 

8.095 

Proportional  variance 
explained 

36% 

34% 

27% 

Cumulative  variance 
explained 

36% 

70% 

97% 

*  indicates  rejected  proposal 


Based  on  these  findings,  the  following  observations  are  made:  The  first  cluster, 
one  whose  proposals  produced  the  largest  value  model  scores  among  the  thirty  proposals, 
would  be  recommended  to  JIEDDO  for  acceptance.  Given  JIEDDO’s  risk  tolerance,  one 
could  determine  whether  it  is  appropriate  it  accept  the  second  cluster  of  proposals  or 
reject  them  or  take  a  closer  look  at  the  members  of  this  group.  The  third  cluster,  those 
with  which  proposals  scored  among  the  lowest  using  the  value  model,  should  be  rejected. 
Thus,  we  would  have  the  tolerance  boundaries  based  on  those  provided  in  Table  8. 
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Table  8:  3  Factor  Analysis  (Rotated  w/Tolerance  Boundaries) 


Proposal  Ranked 

Factor  1 

Factor  2 

Factor  3 

Score 

1 

0.358 

-0.461 

0.711 

0.822 

2 

0.439 

-0.300 

0.849 

0.727 

3* 

0.500 

-0.435 

0.716 

0.683 

4 

0.522 

-0.377 

0.746 

0.672 

5* 

0.521 

-0.443 

0.701 

0.672 

6 

0.598 

-0.459 

0.636 

0.620 

7 

0.718 

-0.350 

0.587 

0.613 

8* 

0.669 

-0.496 

0.523 

0.599 

9 

0.743 

-0.414 

0.517 

0.584 

10 

0.681 

-0.521 

0.491 

0.576 

11 

0.668 

-0.515 

0.525 

0.565 

12 

0.693 

-0.504 

0.492 

0.563 

13* 

0.770 

-0.459 

0.446 

0.561 

14 

0.692 

-0.531 

0.473 

0.561 

15 

0.746 

-0.479 

0.439 

0.555 

16 

0.686 

-0.524 

0.484 

0.554 

17 

0.732 

-0.491 

0.459 

0.539 

18* 

0.687 

-0.541 

0.461 

0.539 

19* 

0.670 

-0.557 

0.463 

0.528 

20* 

0.630 

-0.629 

0.422 

0.502 

21* 

0.597 

-0.636 

0.460 

0.491 

22* 

0.623 

-0.619 

0.443 

0.488 

23* 

0.584 

-0.653 

0.435 

0.477 

24* 

0.518 

-0.703 

0.441 

0.447 

25* 

0.488 

-0.740 

0.457 

0.420 

26* 

0.484 

-0.807 

0.317 

0.401 

27* 

0.452 

-0.827 

0.305 

0.387 

28 

0.447 

-0.804 

0.329 

0.367 

29* 

0.435 

-0.868 

0.252 

0.364 

30* 

0.159 

-0.800 

0.476 

0.170 

Varimax 

10.772 

10.235 

8.095 

Proportional  variance 
explained 

36% 

34% 

27% 

Cumulative  variance 
explained 

36% 

70% 

97% 

*  indicates  rejected  proposal 


For  an  organization  that  is  risk  averse,  proposals  below  the  black  line  would  be 
rejected  by  JIEDDO.  This  would  indicate  a  reject  rate  of  approximately  83.3%.  From 
the  table  above,  we  notice  that  the  majority  of  rejected  proposals  fall  at  or  below  that  of 
proposal  18.  Based  on  the  value  model  results  and  the  above  factor  analysis,  this 
indicates  that  JIEDDO ’s  risk  tolerance  level  may  be  lower  than  that  indicated  by  the 
black  line  in 
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Table  8  above.  Lowering  the  acceptance  boundary  to  include  those  proposals  above  that 
of  proposal  1 8  appears  problematic  because  the  boundary  cuts  right  threw  the  cluster  of 
proposal  6  through  proposal  20.  Specifically,  the  author  acknowledges  that  proposal  20 
through  proposal  22  borders  Factor  one  and  Factor  two.  As  such,  for  an  organization  that 
is  more  risk  tolerant,  it  is  appropriate  to  extend  the  acceptance  line  to  those  proposals 
sitting  above  the  red  line. 

Further  insight  is  gained  after  conducting  cluster  analysis  on  the  set. 


Figure  15:3  Factor  Cluster  Analysis 


The  average  distance  in  the  weight  space  between  proposals  within  a  cluster  is 
smaller  than  those  seen  between  the  clusters  themselves.  From  Figure  15,  we  see  that  the 
second  cluster  contains  the  tightest  proposals;  proposal  7  through  proposal  22  are  on 
average,  a  distance  of  0.032  from  one  another.  Both  clusters  one  and  three  contain 
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proposals  that  are  on  average  approximately  0.080  and  0.074  away  from  each  other, 
respectively.  The  expectation  of  the  cluster  containing  the  highest  scoring  proposals  to 
fall  the  farthest  from  that  of  the  lowest  scoring  proposals  is  demonstrated  here.  Cluster 
one  is  an  average  distance  of  0.241  from  cluster  three.  Cluster  one  and  two  are  the  most 
near  with  an  average  distance  of  0. 1 18.  Finally,  we  see  that  cluster  two  and  cluster  three 
are  approximately  0.135  from  one  another. 

Seeking  to  reduce  the  number  of  rotated  factors,  this  analysis  was  repeated  for 
two  retained  factors  and  is  shown  in  the  tables  below. 


Table  9 


Table  10 


Proposal  Ranked 

Factor  1 

Factor  2 

Score 

1 

0.732 

-0.477 

0.822 

2 

0.902 

-0.327 

0.727 

3* 

0.831 

-0.479 

0.683 

4 

0.872 

-0.424 

0.672 

5* 

0.832 

-0.492 

0.672 

6 

0.832 

-0.526 

0.620 

7 

0.882 

-0.446 

0.613 

8* 

0.790 

-0.583 

0.599 

9 

0.840 

-0.518 

0.584 

10 

0.771 

-0.612 

0.576 

11 

0.789 

-0.601 

0.565 

12 

0.781 

-0.597 

0.563 

13* 

0.801 

-0.571 

0.561 

14 

0.764 

-0.625 

0.561 

15 

0.777 

-0.586 

0.555 

16 

0.769 

-0.616 

0.554 

17 

0.782 

-0.595 

0.539 

18* 

0.751 

-0.635 

0.539 

19* 

0.740 

-0.646 

0.528 

20* 

0.677 

-0.712 

0.502 

21* 

0.684 

-0.710 

0.491 

22* 

0.689 

-0.700 

0.488 

23* 

0.655 

-0.726 

0.477 

24* 

0.612 

-0.762 

0.447 

25* 

0.602 

-0.791 

0.420 

26* 

0.489 

-0.864 

0.401 

27* 

0.457 

-0.879 

0.387 

28 

0.474 

-0.853 

0.367 

29* 

0.402 

-0.919 

0.364 

30* 

0.397 

-0.785 

0.170 

Varimax 

15.822 

12.718 

Proportional  variance 
explained 

53% 

42% 

Cumulative  variance 
explained 

53% 

95% 

Proposal  Ranked 

Factor  1 

Factor  2 

Score 

1 

0.732 

-0.477 

0.822 

2 

0.902 

-0.327 

0.727 

3* 

0.831 

-0.479 

0.683 

4 

0.872 

-0.424 

0.672 

5* 

0.832 

-0.492 

0.672 

6 

0.832 

-0.526 

0.620 

7 

0.882 

-0.446 

0.613 

8* 

0.790 

-0.583 

0.599 

9 

0.840 

-0.518 

0.584 

10 

0.771 

-0.612 

0.576 

11 

0.789 

-0.601 

0.565 

12 

0.781 

-0.597 

0.563 

13* 

0.801 

-0.571 

0.561 

14 

0.764 

-0.625 

0.561 

15 

0.777 

-0.586 

0.555 

16 

0.769 

-0.616 

0.554 

17 

0.782 

-0.595 

0.539 

18* 

0.751 

-0.635 

0.539 

19* 

0.740 

-0.646 

0.528 

20* 

0.677 

-0.712 

0.502 

21* 

0.684 

o 

Cj 

o 

0.491 

22* 

0.689 

-0.700 

0.488 

23* 

0.655 

-0.726 

0.477 

24* 

0.612 

-0.762 

0.447 

25* 

0.602 

-0.791 

0.420 

26* 

0.489 

-0.864 

0.401 

27* 

0.457 

-0.879 

0.387 

28 

0.474 

-0.853 

0.367 

29* 

0.402 

-0.919 

0.364 

30* 

0.397 

-0.785 

0.170 

Varimax 

15.822 

12.718 

Proportional  variance 
explained 

53% 

42% 

Cumulative  variance 
explained 

53% 

95% 
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Figure  16:  2  Factor  Cluster  Analysis 

The  two-factor  analysis  produced  similar  results  as  those  seen  for  the  three-factor 
cluster  analysis.  Cluster  one  above  contains  the  combined  proposals  found  in  clusters 
one  and  two  shown  in  Figure  15:3  Factor  Cluster  Analysis,  proposal  1  through  proposal 
22.  Cluster  two  above  is  analogous  that  that  of  the  third  cluster  in  the  three  factor  cluster 
analysis.  From  the  above,  we  see  that  cluster  one  contains  proposals  that  are  an  average 
of  0.071  from  each  other  within  the  cluster.  Similarly,  cluster  two  contains  proposals  that 
are  0.074  from  one  another.  While  each  cluster  one  and  cluster  two  exhibit  similar 
characteristics  within  the  clusters  themselves,  we  recognize  that  the  distance  between  the 
two  groupings  themselves  is  relatively  far,  0.241.  This  large  distance  implies  that  there 
exists  a  distinct  difference  between  those  proposals  that  comprise  each  group. 

Recognizing  that  the  presence  of  the  R&D  proposals  within  the  factor  analysis 
may  affect  the  clusters,  it  is  important  to  investigate  which  of  the  25  proposals  load  on 
the  new  factors.  Rotated  factor  analysis  was  conducted  on  the  new  set  of  25  proposals. 


62 


When  three  factors  were  retained,  all  of  the  proposals  loaded  on  the  first  two  factors.  As 
a  result,  the  two  rotated  factor  analysis  is  provided  in  Table  11. 


Table  11:  Rotated  Factor  Analysis  (Reduced  Set) 


Proposal  Ranked 

Factor  1 

Factor  2 

Score 

2 

0.880 

0.330 

0.727 

4 

0.848 

0.429 

0.672 

6 

0.826 

0.519 

0.620 

7 

0.896 

0.419 

0.613 

10 

0.805 

0.577 

0.576 

11 

0.803 

0.580 

0.565 

12 

0.805 

0.569 

0.563 

13* 

0.828 

0.538 

0.561 

14 

0.787 

0.599 

0.561 

15 

0.817 

0.545 

0.555 

16 

0.795 

0.588 

0.554 

17 

0.815 

0.560 

0.539 

18* 

0.778 

0.605 

0.539 

19* 

0.769 

0.617 

0.528 

20* 

0.711 

0.682 

0.502 

21* 

0.708 

0.687 

0.491 

22* 

0.721 

0.671 

0.488 

23* 

0.677 

0.706 

0.477 

24* 

0.627 

0.749 

0.447 

25* 

0.622 

0.777 

0.420 

26* 

0.504 

0.857 

0.401 

27* 

0.484 

0.865 

0.387 

28 

0.490 

0.846 

0.367 

29* 

0.406 

0.922 

0.364 

30* 

0.395 

0.795 

0.170 

Eigen  values 

13.198 

10.810 

Proportional  variance 

53% 

43% 

Cumulative  variance 

53% 

96% 

The  reduced  set  contains  25  proposals  verses,  the  original  30.  From  the  table 
above  we  see  that  our  boundary  is  set  at  proposal  22*  for  this  new  data  set.  This 
observed  boundary  location  is  the  same  as  that  which  was  set  for  the  rotated  three-factor 
analysis.  As  such,  we  suggest  that  proposals  that  fall  at  or  above  that  of  proposal  22* 
exhibit  similar  patterns  in  behavior.  There  are  1 7  proposals  that  rank  at  or  above  that 
identified  by  proposal  22*.  Proposals  23*  through  30*  comprise  the  remaining  eight 
proposals.  We  would  expect  the  top  ranked  alternatives,  the  first  17,  to  be  accepted.  This 
analysis  supports  the  rejection  of  proposals  falling  below  proposal  22*. 
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Figure  17:  2  Factor  Cluster  Analysis  for  Reduced  Set 

The  patterns  exhibited  in  Figure  17  above  nearly  mirror  those  seen  for  the 
two-factor  cluster  analysis  conducted  on  all  30  proposals.  However,  the  distinction 
between  the  full  and  reduced  set  is  captured  in  the  first  cluster.  The  distances  between 
proposals  within  this  set  are  approximately  0.055,  while  those  seen  for  the  second  cluster 
are  0.074.  While  the  distance  between  the  two  clusters  in  the  original  set  was 
approximately  0.241,  the  average  distance  between  the  two  clusters  for  the  reduced  set  is 
smaller,  0.153. 

In  addition  to  observing  the  clustering  affect  among  proposals,  it  is  important  to 
investigate  the  weight  changes  themselves  to  provide  a  more  extensive  comparison  of  the 
proposals.  All  of  the  weights  that  comprise  each  cell  of  the  distance  matrix  were 
recorded  for  their  respective  proposal.  From  these  weight  values,  it  was  possible  to 
record  the  percent  weight  change  that  occurred  from  the  original  global  weight  to  the  new 
weight  that  minimized  the  L2  norm  for  this  specific  problem.  The  images  provided 
depicts  the  percent  weight  change  as  well  as  the  actual  weight  changes  that  occurred 
among  proposals  that  ranked  among  the  highest  in  the  set. 


64 


The  first  set  of  two  figures  describes  the  percent  and  actual  weight  changes  for  the 
first  proposal.  The  images  show  the  weights  sorted  from  the  largest  original  global 
weight  value  (Gap  Impact)  to  the  least  original  global  weight  value  (Program  Maturity) 


for  both  percent  change  and  actual  weight  change  images. 


Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-42  -35  81  -34  -46  -4  79  64  -10  33  -30  -40  306 


*  Indicates  JIEDDO  rejected  proposal 


Figure  18:  Proposal  #1  Percent  Change 
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Proposal  Ranked  # 
1 
2 
3* 

4 

5* 

6 

7 

8* 

9* 

10 

11 

12 

13* 

14 

15 

16 
17 
18* 

19* 

20* 

21* 

22* 

23* 

24* 

25* 

26* 

27* 

28 

29* 

30* 

Original  Global 
Weights 

Average  Weight 
Change 


*  Indicates  JIEDDO  rejected  proposal 


Percentile 


I 

I 


100 

95 

90 

85 

80 

75 

70 

65 

60 

55 

50 

45 

40 

35 

30 

25 

20 

15 

10 

5 

0 


Weight  Change 
0.165 
0.102 
0.066 
0.051 
0.044 
0.036 
0.031 
0.027 
0.027 
0.021 
0.014 
0.004 
0.000 
-0.016 
-0.034 
-0.037 
-0.054 
-0.069 
-0.089 
-0.106 
-0.176 


Figure  19:  Proposal  #1  Weight  Change 


Figure  19  illustrates  that  the  average  weight  change  across  all  30  proposals  for 
Gap  Impact  is  -0.074,  while  the  average  weight  change  for  Program  Maturity  is  0.040. 
Sorting  the  data  by  average  weight  change,  from  most  positive  to  most  negative,  for  each 
of  the  thirteen  values,  allows  us  to  see  which  value,  on  average,  changes  the  most. 
Additionally  the  “Sorted  Vertically”  metrics  describe  the  same  set  of  information  as  those 
mentioned  above;  however,  the  data  has  been  sorted  by  the  average  weight  change  for 
each  value. 

We  see  from  the  average  weight  change  images  shown  in  Figure  20  and  Figure  2 1 
that  Program  Maturity,  Technical  Perfonnance,  Tenets  Impacted,  and  Classification  need 
to  increase  simultaneously  in  order  for  some  rank  order  pressure  to  affect  that  of  the  top 
ranked  alternative.  Meanwhile,  the  weight  for  Gap  Impact  needs  to  decrease.  This 
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indicates  that  proposal  one  scored  very  well  in  meeting  the  Gap  Impact,  and  relatively 


poor  for  Program  Maturity,  Technical  Performance,  Tenets  Impacted,  as  well  as 
Classification. 


Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.013  0.110  0.056  0.056  0.056  0.087  0.056  0.050  0.100  0.112  0.037  0.176  0.091 


306  81  79  64  33  -4  -10  -30  -34  -35  -40  -42  -46 


■  Indicates  JIEDDO  rejected  proposal 


Figure  20:  Proposal  #1  Percent  Change  (Sorted  Vertically) 


Proposal  Ranked  # 


Original  Global 
Weights 


Average  Weight 
Change 


0.110  0.056  0.013  0.056  0.056  0.087  0.056  0.037  0.050  0.100  0.112  0.091  0.176 


0.089  0.044  0.040  0.036  0.019  -0.003  -0.006  -0.015  -0.015  -0.034  -0.039  -0.042  -0.074 


*  Indicates  JIEDDO  rejected  proposal 


Figure  21:  Proposal  #1  Weight  Change  (Sorted  Vertically) 
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We  expect  that  significant  weight  changes  need  to  occur  for  a  rank  reversal  to 
occur  among  proposals  that  scored  among  the  highest  and  lowest  for  this  evaluation. 

This  suspicion  is  supported  by  the  previously  produced  images.  We  see  that  the  red  and 
green  suggests  both  significant  weight  increases  and  decreases  for  rank  changes  to  occur. 
This  phenomenon  is  observed  for  all  proposals  in  the  proposal  one  example.  That  is,  as 
the  distance  between  proposals  increases,  the  ability  for  a  rank  change  to  occur  is 
dependent  on  larger  positive  or  negative  simultaneous  weight  changes.  All  of  the  images 
generated  for  this  percent  weight  change  analysis  are  provided  in  Appendix  C. 

A  more  interesting  area  of  interest  is  those  proposals  that  fall  in  the  mid-range  for 
JIEDDO  proposals.  From  this,  the  following  question  surfaces.  Given  that  we  know 
proposals  that  score  highly  should  be  accepted,  what  conclusions  could  be  drawn  from 
those  that  score  in  the  second  and  third  clusters  for  value  scoring?  To  examine  this 
further,  we  extract  percent  change  images  for  two  proposals  that  line  in  this  mid  range. 
We  begin  with  analysis  for  the  previously  evaluated  Proposal  P*,  the  13th  ranked 
proposal. 
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i  * 


Proposal 
Ranked  # 


: 


Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-9.10  -7.47  5.20  -6.27  -16.57  -11.10  22.20  15.30  8.37  5.80  21.50  -7.03  95.33 


*  Indicates  JIEDDO  rejected  proposal 


r 

I 
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Figure  22:  Proposal  #13  Percent  Change 


Proposal  Ranked 


1 1 
si 


I* 


Original  Global 
Weights 


Average  weight 
change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


0.006  -0.006  -0.015  -0.010  0.003  0.013  0.005  0.008  0.011  -0.003  0.012 


I 

I 


Percentile  Weight  Change 

0.154 
0.053 
0.031 
0.018 
0.010 


0.004 

0.002 

0.001 

0.001 

0.000 

0.000 

0.000 

-0.001 

-0.003 

-0.007 

-0.010 

-0.017 

-0.030 

-0.054 

-0.176 


*  Indicates  JIEDDO  rejected  proposal 


Figure  23:  Proposal  #13  Weight  Change 
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From  the  images  of  proposal  13,  we  see  a  light  band  of  color  between  proposals 


1 1  to  14  with  some  mild  shading  that  occurs  just  beyond  that  region.  In  both  of  the 


transformed  images  shown  below,  we  observe  the  sensitivity  of  small  percent  (or  actual 


weight)  changes  to  rank  change.  The  percent  change  in  this  band  is  observed  to  lie 


between  one  and  negative  two  percent.  This  amounts  to  an  actual  weight  change  of  +/- 


.0001.  We  see  the  potential  for  inconsistency  in  choosing  to  reject  a  proposal  like  13, 


whose  immediate  neighbors  12  and  14,  were  accepted.  Minimal  if  no  changes  are 


required  among  the  values  for  rank  changes  to  occur. 


Proposal 
Ranked  # 


3* 


8* 

9* 

10 

11 

12 

13* 

14 

15 

16 
17 
18* 
19’ 
20* 
21* 
22* 
23* 
24* 
25* 
26* 
27* 
28 
29* 
30* 


Original 

Global 

Weights 


Average  % 
Weight 
Change 


0.01  0.06  0.05  0.06  0.06  0.06  0.11  0.10  0.04  0.11  0.18  0.09  0.09 


95.33  22.20  21.50  15.30  8.37  5.80  5.20  -6.27  -7.03  -7.47  -9.10  -11.10  -16.57 


*  Indicates  JIEDDO  rejected  proposal 
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Figure  24:  Proposal  #13  Percent  Change  (Sorted  Vertically) 
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Proposal  Ranked 
# 

1 

2 

3* 


5* 

6 

7 

8* 

9* 

10 

11 

12 

13’ 

14 

15 

16 
17 
18’ 

19’ 

20’ 

21* 

22’ 

23* 

24’ 

25* 

26’ 

27* 

28 

29’ 

30’ 

Original  Global 
Weights 

Average  weight 
change 


0.056  0.013  0.050  0.056  0.110  0.056  0.056  0.037  0.100  0.112  0.087  0.091  0.176 


0.013  0.012  0.011  0.008  0.006  0.005  0.003  -0.003  -0.006  -0.008  -0.010  -0.015  -0.016 


*  Indicates  JIEDDO  rejected  proposal 


Figure  25:  Proposal  #13  Weight  Change  (Sorted  Vertically) 


Continuing  this  investigation,  we  see  the  same  shading  ban  appear  for  proposal 
13’s  neighbor.  We  see  from  the  percent  change  and  weight  change  metric,  that  proposal 
14  scored  relatively  well  for  Gap  Impact,  Technical  Performance,  Classification,  Fielding 
Timeline,  Technical  Risk,  and  Program  Maturity.  However,  indications  of  poor  ratings 
arise  for  Time  to  Counter,  Work  Load,  Interoperability,  Suitability  Level,  as  well  as 
Training  Timeline.  These  patterns  are  exhibited  in  Figure  26  through  Figure  29. 
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Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.05  0.037  0.013 


-3.800  -11.133  7.867  0.500  -10.633  -5.033  6.867  12.167  6.267  9.133  3.967  -11.900  52.667 


*  Indicates  JIEDDO  rejected  proposal 


Figure  26:  Proposal  #14  Percent  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.05  0.037  0.013 


-4  -11  8  1  -11  -5  12  9  7  6  4  -12  53 


*  Indicates  JIEDDO  rejected  proposal 


Figure  27:  Proposal  #14  Weight  Change 
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Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.013  0.056  0.056  0.110  0.056  0.056  0.05  0.100  0.176  0.087  0.091  0.112  0.037 


53  12  9  8  7  6  4  1  -4  -5  -11  -11  -12 


*  Indicates  JIEDDO  rejected  proposal 


Figure  28:  Proposal  #14  Percent  Change  (Sorted  Vertically) 
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Figure  29:  Proposal  #14  Weight  Change  (Sorted  Vertically) 
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Now  that  we  have  made  observations  of  the  sensitivity  of  the  top  two  thirds  of  the 
provided  JIEDDO  proposals,  it  is  necessary  to  turn  to  the  bottom  third.  For  completeness 
sake,  one  proposal  will  be  selected  as  a  representative  of  the  set,  for  discussion.  The 
expected  observations  for  this  set  would  resemble  those  observed  among  proposals  in  the 
top  third.  More  specifically,  we  would  expect  to  see  significant  positive  and  negative 
weight  changes  (as  indicated  by  the  green  and  red  shading  intensity)  in  order  for  a 
proposal  that  is  ranked  relatively  low  to  swap  places  with  one  of  its  competitors. 

Proposal  26  was  randomly  selected  among  the  group  for  observation. 


Original 

Global 

Weights 


Average  % 
Weight 
Change 


Figure  30:  Proposal  #26  Percent  Change 


0.18  0.11  0.11  0.10  0.09  0.09  0.06  0.06  0.06  0.06  0.05  0.04  0.01 


-20.40  27.53  10.70  7.43  -37.47  -36.60  62.37  39.40  13.63  -10.57  -37.97  -23.47  160.47 


*  Indicates  JIEDDO  rejected  proposal 
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Proposal  Ranked 
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Original  Global 
Weights 

Average  weight 
change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.036  0.031  0.012  0.007  -0.034  -0.032  0.035  0.022  0.008  -0.006  -0.019  -0.009  0.021 


*  Indicates  JIEDDO  rejected  proposal 
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Figure  3 1 :  Proposal  #26  Weight  Change 
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Figure  32:  Proposal  #  26  Percent  Change 


0.01  0.06  0.06  0.11  0.06  0.11  0.10  0.06  0.18  0.04  0.09  0.09  0.05 


160.47  62.37  39.40  27.53  13.63  10.70  7.43  -10.57  -20.40  -23.47  -36.60  -37.47  -37.97 


*  Indicates  JIEDDO  rejected  proposal 
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Proposal  Ranked 
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Original  Global 
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Average  weight 
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j  0.056  0.112  0.056  0.013  0.110  0.056  0.100  0.056  0.037  0.050  0.087  0.091  0.176 


0.035  0.031  0.022  0.021  0.012  0.008  0.007  -0.006  -0.009  -0.019  -0.032  -0.034  -0.036 


*  Indicates  JIEDDO  rejected  proposal 


Figure  33:  Proposal  #26  Weight  Change 


As  predicted,  we  see  intense  positive  (green)  and  negative  (red)  shading  in  the 
figures  above.  The  new  weights  produced  from  the  optimization  problem  suggest 
simultaneously  increasing  the  values  for  Tenets  Impacted,  Time  to  Counter,  and  Program 
Maturity.  Conversely,  we  see  weight  decreases  for  Training  Time,  Operations  Burden, 
Interoperability,  and  Gap  hnpact.  Technical  Performance,  Suitability,  Work  Load, 
Fielding  Timeline,  and  Technical  Risk  show  indications  of  positive  and  negative  changes 
as  proposals  comparisons  are  made  between  proposal  26  and  the  highest  ranked 
proposals. 

IV.E  Summary 

The  purpose  of  this  chapter  is  to  provide  a  clear  and  concise  summary  of  the 
JIEDDO  model  validation  process  as  well  as  the  sensitivity  analysis  procedures  thereof. 
From  this  result,  this  research  proposed  a  method  for  evaluating  proposals  using 
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discriminant  analysis.  More  specifically,  given  13  distinct  considerations  (values)  and  a 
known  accept  or  reject  outcome,  we  were  able  to  develop  a  technique  that  mirrors 
JIEDDO’s  decision  process.  Given  a  new  set  of  proposals,  we  would  be  able  to 
accurately  predict  whether  a  panel  of  decision  makers  is  likely  to  accept  or  reject  the 
proposal.  Once  this  insight  was  gained,  the  second  part  of  the  chapter  was  dedicated  to 
the  sensitivity  of  the  JIEDDO  value  model  global  weights  within  the  decision  problem. 
From  this,  the  research  proposed  a  new  sensitivity  analysis  imaging  technique  utilizing 
known  math  programming  and  multivariate  techniques. 
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V.  Conclusions  and  Recommendations 


V.A.  Introduction 

Military  members  serving  in  Iraq  and  Afghanistan  face  many  challenges  as  they 
persist  in  their  counter  terrorism  mission.  Insurgents  are  undoubtedly  dedicated  to  using 
unconventional  means,  particularly  IEDs,  to  defeat  the  Allied  coalition.  We  have  seen 
military  members  serving  in  Iraq  and  Afghanistan,  in  particular,  require  a  great  deal  of 
support  from  both  the  Department  of  Defense  as  well  as  the  American  people  as  they 
continue  to  face  and  overcome  obstacles.  In  response  to  this  need,  JIEDDO  has  served  as 
a  lead  organization  in  the  solicitation  and  development  of  C-IED  projects.  Their  primary 
motivation  is  centered  on  answering  the  warfighter’s  urgent  need  by  delivering  the  most 
appropriate  C-IED  systems  as  rapidly  as  possible. 

We  have  seen  how  Decision  Analysis  has  the  potential  to  serve  JIEDDO  in 
critically  evaluating  and  selecting  C-IED  proposals.  The  development  of  a  Value 
Focused  Thinking  model  that  properly  identifies  JIEDDO  values  as  a  basis  for  proposal 
evaluation  will  serve  as  an  appropriate  mechanism  for  evaluating  the  organization’s 
ability  to  meet  the  warfighters  perceived  needs.  The  criticality  of  verifying  and 
validating  the  JIEDDO  value  model’s  robustness  is  imminent  in  cases  where  lives  are  at 
stake. 

V.B.  Research  Contributions 

The  purpose  of  this  research  was  to  provide  contributions  to  the  field  of  Decision 
Analysis  in  the  areas  of  model  validation  and  sensitivity  analysis.  This  research  showed 
how  Decision  Analysis  and  Discriminant  Analysis  techniques  can  be  merged  to  provide 
insight  into  decision  outcome. 
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The  first  contribution  is  in  the  area  of  model  validation.  This  research  applied 
Discriminant  Analysis  techniques  to  the  model  as  a  means  to  develop  a  function  that 
describes  a  particular  decision  process.  As  such,  this  research  showed,  by  means  of  the 
Lachenbruch  Holdout  Procedure,  that  it  is  possible  to  utilize  value  model  attributes  to 
form  a  discriminant  function  to  describe  the  proposal  evaluation  process.  This  procedure 
is  useful  when  utilizing  value  focused  thinking  models,  like  JIEDDO,  to  repeatedly  make 
a  decision. 

The  second  contribution  to  the  field  of  DA  is  in  the  area  of  boundary 
detennination.  For  decision  models  where  multiple  alternatives  are  selected,  it  is 
important  to  identify  at  what  point  should  the  line  be  drawn  for  selecting  a  given 
alternative  verses  rejecting  it.  Factor  Analysis  serves  as  a  useful  technique  for 
detennining  the  acceptance-rejection  boundary.  The  correlation  between  proposals  in  the 
factors  themselves  allows  us  to  identify  clusters  of  proposals  that  are  considered  very 
similar.  As  a  result,  those  proposals  that  are  clustered  are  considered  too  similar  to  reject 
one  while  accepting  the  others. 

The  third  contribution  to  the  field  of  DA  is  in  the  area  of  Sensitivity  Analysis. 
Prior  research  demonstrated  the  usefulness  of  using  math  programming  techniques  to 
detennine  various  distances  between  alternatives.  This  research  utilized  math 
programming  techniques  to  develop  an  image  profiling  tool  to  evaluate  proposals. 
Applying  this  technique  allows  decision  makes  to  understand  how  well  a  specific 
proposal  prefonned  against  each  one  of  its  competitors. 
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V.C.  Recommendations  for  Further  Research 


Sensitivity  Analysis  remains  a  trusted  mechanism  for  determining  the  level  of 
confidence  a  DM  can  have  in  the  problem  they  are  looking  to  solve.  Decision  Analysts 
recognize  that  much  of  sensitivity  analysis  involves  observations  in  small  or  even  large 
fluctuations  among  the  weights  themselves.  We  have  seen  how  the  implication  of  weight 
uncertainty  leads  us  to  pose  the  following  questions,  “Are  the  values  weighted 
appropriately  in  a  manner  that  truly  reflects  the  decision  maker’s  preference?  If  there 
exists  error  among  the  weights,  how  much  or  little  error  is  there?  How  will  it  affect  the 
decision?”  The  subjectivity  of  the  weight  elicitation  process  remains  at  the  root  of 
concerns. 

However,  instead  of  taking  a  forward  approach  to  eliciting  weights  to  calculate 
the  value  score  for  a  particular  decision  problem,  it  would  be  interesting  to  investigate  a 
backward  approach  to  extract  weights.  Given  the  values  and  an  outcome  (i.e.  accept  or 
reject  as  seen  in  JIEDDO),  is  it  possible  to  determine  the  set  of  weights  that  would 
produce  such  a  known  outcome?  The  answer  to  this  question  could  be  approached 
utilizing  discriminant  analysis  techniques.  The  idea  being  simple,  using  historical  data  to 
extract  the  set  of  variables  and  known  outcome,  it  is  possible  to  create  a  discriminant 
function  that  will  define  each  population  of  interest.  This  will  allow  us  to  determine  the 
weights  that  a  decision  maker  places  on  each  of  the  respective  value  categories.  It  is  then 
possible  to  compare  the  weights  produced  using  this  weight  extraction  procedure  to  those 
elicited  from  the  DM.  However,  the  need  for  access  to  historical  data  is  imminent  for  this 
backward  weight  extraction  to  work.  This  concept  will  serve  beneficial  for  decision 
problems  whereby  a  decision  is  made  more  than  once. 
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V.D.  Conclusions 


Decision  Analysis  models  are  useful  tools  for  analyzing  various  alternatives 
available  in  order  to  make  the  most  sound  decision  for  the  problem  at  hand.  Value 
Focused  Thinking  allows  a  team  of  analysts  to  work  with  a  decision  maker  to  identify  the 
values  for  the  decision,  weight  the  values  appropriately,  and  evaluate  alternatives  based 
on  the  aforementioned  criteria.  Recognizing  that  the  subjectivity  of  weight  elicitation 
serves  as  motivation  for  model  verification  and  validation,  this  research  demonstrated  the 
practicality  and  usefulness  of  applying  Discriminant  Analysis  techniques  to  verify  the 
consistency  between  the  decision  maker’s  decision  and  the  model’s  recommendation. 
Additionally,  we  recognize  the  usefulness  of  checking  weight  sensitivity  via  an  image 
profiling  technique.  Imaging  serves  as  a  means  for  evaluating  a  given  alternative  to  see 
how  it  stands  as  ranked  against  its  competitors. 

The  application  of  such  research  topics  is  promising  for  the  JIEDDO  proposal 
evaluation  process.  There  is  no  question  as  to  the  importance  of  evaluating  and  selecting 
the  best  C-IED  proposals  to  meet  the  current  warfighter’s  needs.  Appropriately 
developed  and  validated  value  models  aid  decision  makers,  like  JIEDDO,  in  making 
decisions  that  will  accommodate  our  military  service  members  seeking  to  overcome 
obstacles  in  Iraq  and  Afghanistan. 
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Appendix 


Appendix  A:  JIEDDO  Proposal  Data 


Alternative  I  #  Tenets  Primary  Gap  Classification  Months  Useful  Performance 


Suitability  Interop.  Issues 


Significant 


Alternative 


Months  to  Fielding  %  Max  Capacity  Interaction  Min/Hr  Training  Hours  Training  Level 


Appendix  C:  Proposal  Rank  Comparison  Images 


Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-42  -35  81  -34  -46  -4  79  64  33  -10  -30  -40  306 


*  Indicates  JIEDDO  rejected  proposal 


Figure  34:  Proposal  #1  Percent  Change 
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Proposal  Ranked  # 


Original  Global 
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Average  Weight 
Change 


0.176  0.112  0.110  0.100 


0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.074  -0.039  0.089  -0.034  -0.042  -0.003  0.044  0.036  0.019  -0.006  -0.015  -0.015  0.0401 


*  Indicates  JIEDDO  rejected  proposal 
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0.000 
-0.016 
-0.034 
-0.037 
-0.054 
-0.069 
-0.089 
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Figure  35:  Proposal  #2  Percent  Change 
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Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.57  -17.67  -8.10  -29.57  -46.50  -8.50  82.37  39.37  -9.13  -14.73  43.77  -43.57  372.43 


'  Indicates  JIEDDO  rejected  proposal 
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Figure  36:  Proposal  #2  Percent  Change 
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0.176  0.112  0.110  0.100 


0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


Average  Weight  I  -0.001  -0.020  -0.009  -0.030  -0.042  -0.007  0.046  0.022  -0.005  -0.008  0.022  -0.016  0.0491 
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*  Indicates  JIEDDO  rejected  proposal 
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Weight  Change 
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0.029 
0.020 
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0.005 
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Figure  37:  Proposal  #3  Weight  Change 
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Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-8  -22  37  -21  -32  -14  27  24  -14  60  -29  16  120 


*  Indicates  JIEDDO  rejected  proposal 


Figure  38:  Proposal  #3  Percent  Change 


Figure  39:  Proposal  #3  Weight  Change 
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Global 
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Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-13  -32  25  12  -7  -6  -7  39  12  31  -36  -44  176 


’  Indicates  JIEDDO  rejected  proposal 


Figure  40:  Proposal  #4  Percent  Change 
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Figure  41:  Proposal  #4  Weight  Change 
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Figure  42:  Proposal  #5  Percent  Change 
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*  Indicates  JIEDDO  rejected  proposal 
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*  Indicates  JIEDDO  rejected  proposal 


Figure  43:  Proposal  #5  Weight  Change 
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Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.18  0.11  0.11  0.10  0.09  0.09  0.06  0.06  0.06  0.06  0.05  0.04  0.01 


-7.37  -20.93  10.27  17.27  -9.10  -6.33  -22.40  15.57  25.80  0.80  28.47  -34.00  70.30 


*  Indicates  JIEDDO  rejected  proposal 


Figure  44:  Proposal  #6  Percent  Change 
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Original  Global 
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Average  weight 
change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


0.013  -0.024  0.011  0.017  -0.008  -0.006  -0.013  0.009  0.015  0.000  0.014  -0.013  0.009 


'  Indicates  JIEDDO  rejected  proposal 


Figure  45:  Proposal  #7  Weight  Change 
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3.1 76  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-10  -16  16  -16  -15  -16 


35  8  -9  -6  13  156 


*  Indicates  JIEDDO  rejected  proposal 


Figure  46:  Proposal  #7  Percent  Change 
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Average  weight  I  -0.018  -0.0' 
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0.017  -0.016  -0.013  -0.014  0.020  0.01 
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*  Indicates  JIEDDO  rejected  proposal 


Figure  47:  Proposal  #7  Weight  Change 
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0.01  0.04  0.06  0.06  0.11  0.06  0.18  0.06  0.09  0.05  0.10  0.11  0.09 


61.10  24.57  15.00  14.83  13.73  10.13  -0.83  -4.60  -8.27  -9.13  -9.67  -12.30  -16.53 


*  Indicates  JIEDDO  rejected  proposal 


Figure  48:  Proposal  #8  Percent  Change 
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Proposal  Ranked  ^ 


Original  Global 
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Average  weight 
change 


0.176  0.112  0.110  0.100 


0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.001  -0.014  0.015  -0.010  -0.015  -0.007  0.006  0.008  -0.003  0.008  -0.005  0.009  0.01 


*  Indicates  JIEDDO  rejected  proposal 
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Figure  49:  Proposal  #8  Weight  Change 


92 


Original 

Global 

Weights 
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Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-4  0  11  -6  -23  -10  -4  31  -9  18  0  -22  136 


*  Indicates  JIEDDO  rejected  proposal 


Figure  50:  Proposal  #9  Percent  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.007  0.000  0.012  -0.006  -0.021  -0.008  -0.002  0.018  -0.005  0.010  0.000  -0.008  0.0181 


*  Indicates  JIEDDO  rejected  proposal 
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Figure  5 1 :  Proposal  #9  Weight  Change 
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Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-3.57  -9.83  10.90  -6.77  -12.20  -6.17  14.60  13.03  -2.23  12.10  -5.70  8.60  54.30 


*  Indicates  JIEDDO  rejected  proposal 


Figure  52:  Proposal  #10  Percent  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


0.006  -0.011  0.012  -0.007  -0.011  -0.005 


0.007  -0.001  0.007  -0.003  0.003  0.007 


*  Indicates  JIEDDO  rejected  proposal 
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Figure  53:  Proposal  #10  Weight  Change 
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Global 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-6.53  -3.30  1.63  3.43  -10.10  -5.77  9.83  9.67  11.70  5.57  11.47  -21.20  43.50 


*  Indicates  JIEDDO  rejected  proposal 


Figure  54: 


Proposal  #11  Percent  Change 
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*  Indicates  JIEDDO  rejected  proposal 
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Figure  55:  Proposal  #1 1  Weight  Change 
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Global 
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Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-5.83  -12.17  10.30  0.87  -12.83  -6.13  13.40  13.13  9.03  15.50  -12.37  -4.03  58.67 


*  Indicates  JIEDDO  rejected  proposal 


Figure  56:  Proposal  #12  Percent  Change 
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*  Indicates  JIEDDO  rejected  proposal 
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Figure  53:  Proposal  #12  Weight  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-9.10  -7.47  5.20  -6.27  -16.57  -11.10  22.20  15.30  8.37  5.80  21.50  -7.03  95.33 


*  Indicates  JIEDDO  rejected  proposal 


Figure  58: 


Proposal  #13  Percent  Change 
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*  Indicates  JIEDDO  rejected  proposal 
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Figure  59:  Proposal  #13  Weight  Change 
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J.176  0.112  0.110  0.100 


0.087  0.056  0.056  0.056  0.056 


0.037  0.013 


-4  -11  8  1  -11  -5  12 


"  Indicates  JIEDDO  rejected  proposal 


Figure  60:  Proposal  #14  Percent  Change 


Percentile 

1100 
S5 
90 
85 
80 

s 

65 
60 
55 
50 
45 
40 
35 
30 
25 
20 
15 
10 


Percent  Change 

(%) 


Proposal  Ranked 
# 


3* 


8* 

9* 

10 

11 

12 

13* 

14 

15 

16 
17 
18’ 

19* 

20* 

21* 

22* 

23* 

24’ 

25* 

26’ 

27* 

28 

29* 

30* 

Original  Global 
Weights 


Average  weight 
change 


*  Indicates  JIEDDO  rejected  proposal 


Figure  61:  Proposal  #14  Weight  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-2  -7  10  -12  -19  -14  25  19  8  -4  1  1  106 


’  Indicates  JIEDDO  rejected  proposal 


Figure  62: 


Proposal  #15  Percent  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.003  -0.008  0.011  -0.012  -0.017  -0.012  0.014  0.010  0.005  -0.002  0.000  0.000  0.014 


*  Indicates  JIEDDO  rejected  proposal 


Figure  63:  Proposal  #15  Weight  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 
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Figure  64:  Proposal  #16  Percent  Change 


Figure  65:  Proposal  #16  Weight  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


1.77  -9.90  3.37  -3.03  -11.03  -10.77  15.00  22.20  -4.67  -0.37  -3.63  -4.13  93.67 


*  Indicates  JIEDDO  rejected  proposal 


Figure  66:  Proposal  #17  Percent  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


0.003  -0.011  0.004  -0.003  -0.010  -0.009  0.008  0.012  -0.003  0.000  -0.002  -0.001  0.012 


*  Indicates  JIEDDO  rejected  proposal 


Figure  67:  Proposal  #17  Weight  Change 
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Figure  68:  Proposal  #18  Percent  Change 
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*  Indicates  JIEDDO  rejected  proposal 


Figure  69:  Proposal  #18  Weight  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-2.57  -9.90  6.10  -2.33  -11.57  -5.57  11.83  14.67  6.30  13.73  -0.57  -17.00  55.70 


Figure  70:  Proposal  #19  Percent  Change 
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*  Indicates  JIEDDO  rejected  proposal 


Figure  71:  Proposal  #19  Weight  Change 
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Figure  72:  Proposal  #20  Percent  Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-4.00  -14.20  12.40  -4.33  -12.03  -1.13  -8.07  13.97  10.63  2.70  0.63  19.33  55.97 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.007  -0.016  0.014  -0.004  -0.011  -0.001  -0.005  0.008  0.006  0.002  0.000  0.007  0.007 


'  Indicates  JIEDDO  rejected  proposal 


Figure  73:  Proposal  #20  Weight  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-5.40  -10.80  10.73  -1.77  -15.63  -2.27  5.00  10.27  -5.70  14.27  8.67  13.00  40.77 


Figure  74:  Proposal  #21  Percent  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 
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*  Indicates  JIEDDO  rejected  proposal 


Figure  75:  Proposal  #21  Weight  Change 
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Figure  76:  Proposal  #22  Percent  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-0.004  -0.015  0.009  0.005  -0.013  -0.013  0.006  0.008  0.004  0.006  0.006  -0.006  0.007 


*  Indicates  JIEDDO  rejected  proposal 


Figure  77:  Proposal  #22  Weight  Change 
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0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


3.00  -14.50  6.90  -2.27  -23.23  3.57  25.60  18.37  13.63  4.40  -24.47  -21.83  71.13 


Figure  78:  Proposal  #23  Percent  Change 
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*  Indicates  JIEDDO  rejected  proposal 
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Figure  79:  Proposal  #23  Weight  Change 
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Figure  80:  Proposal  #24  Percent  Change 
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Figure  81:  Proposal  #24  Weight  Change 
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Figure  82:  Proposal  #25  Percent  Change 
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Figure  83:  Proposal  #25  Weight  Change 
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Figure  84:  Proposal  #26  Percent  Change 


Figure  85:  Proposal  #26  Weight  Change 
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Figure  86:  Proposal  #27  Percent  Change 


Figure  87:  Proposal  #27  Weight  Change 


111 


Original 

Global 

Weights 

Average  % 
Weight 
Change 


0.176  0.112  0.110  0.100  0.091  0.087  0.056  0.056  0.056  0.056  0.050  0.037  0.013 


-23.17  -23.63  34.60  -36.50  -2.77  -42.40  17.53  40.00  66.67  0.57  35.73  -8.27  157.07 


Figure  88:  Proposal  #28  Percent  Change 
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Figure  89:  Proposal  #28  Weight  Change 
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Figure  90:  Proposal  #29  Percent  Change 
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Figure  92:  Proposal  #30  Percent  Change 
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